User-defined indexing of multimedia content

• US 20050193005A1
• Filed: 02/13/2004
• Published: 09/01/2005
• Est. Priority Date: 02/13/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Evans, Glenn F., Weiss, Rebecca C., Bradstreet, John, McCullough, Luke Wabaunsee, Gates, Matthijs A.
• Granted Patent: US 7,984,089 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/489 using time information

Abstract

Methods, systems, and computer program products for arbitrarily indexing multimedia content. In accordance with example embodiments, a multimedia content stream that comprises one or more samples is received. At least one of the samples is passed to one or more multimedia filters for analysis based on criteria that define an index type for the multimedia filter. Once it is determined that the sample meets the criteria for an arbitrary filter and index type, an index entry that identifies both the sample and the index type of the arbitrary filter is created. The index entry is adapted in accordance with a common, extensible indexing scheme to coexist with other index entries of other or the same index type for the sample. The created index entry is stored for subsequent access.

Claims

1. In a system capable of processing multimedia content, a method of adding an arbitrary index to the multimedia content such that the arbitrary index is capable of coexisting and not interfering with one or more other arbitrary indexes regardless of index type, the method comprising acts of:

receiving one or more multimedia content streams, each comprising one or more samples;

passing at least one sample to one or more multimedia filters for analysis based on one or more criteria that define an index type for each of the one or more multimedia filters;

determining that the at least one sample meets the one or more criteria for an arbitrary filter and index type;

creating an index entry that identifies the at least one sample and the index type of the arbitrary filter, the index entry being adapted in accordance with a common, extensible indexing scheme to coexist and not interfere with one or more other index entries of either a same or different type for the at least one sample; and

storing the index entry for subsequent access.

2. A method as recited in claim 1, wherein the at least one sample has been indexed with at least one other index.

3. A method as recited in claim 2, wherein the at least one other index is a time-based index.

4. A method as recited in claim 1, wherein the at least one sample has been indexed previously with an index of the same type, which is kept separate from the created and stored index entry.

5. A method as recited in claim 1, further comprising acts of:

determining that the at least one sample meets the one or more criteria for another filter and index type;

creating another index entry that identifies the at least one sample and the index type of the other filter; and

storing the other index entry for subsequent access.

6. A method as recited in claim 1, further comprising acts of:

passing the at least one sample to a plurality of multimedia filters for analysis based on one or more criteria that define an index type for each of the plurality of multimedia filters;

determining that the at least one sample meets the one or more criteria for multiple filters and index types;

creating a plurality of index entries, each identifying the at least one sample and the index type for one of the multiple filters, the index entries being adapted in accordance with a common, extensible indexing scheme to coexist and not interfere with each other; and

storing the plurality of index entries for subsequent access.

7. A method as recited in claim 6, wherein at least two of the multiple multimedia filters are in parallel and are passed the at least one sample simultaneously, and wherein a first of the multimedia filters in parallel has access to analysis performed by a second of the multimedia filters in parallel, the method further comprising the first multimedia filter using the analysis performed by the second multimedia filter in analyzing the at least one sample and determining whether the at least one sample meets the one or more criteria of the first multiple filter.

8. A method as recited in claim 6, wherein a first multimedia filter and a second multimedia filter are in series and are passed the at least one sample sequentially, and wherein the second multimedia filter has access to analysis performed by the first multimedia filter, the method further comprising the second multimedia filter using the analysis performed by the first multimedia filter in analyzing the at least one sample so that the second multimedia filter need not duplicate one or more portions of the analysis performed by the first multimedia filter.

9. A method as recited in claim 1, wherein the one or more multimedia content streams are received from a multimedia content broadcaster, and wherein the arbitrary filter identifies the at least one sample as a new scene.

10. A method as recited in claim 1, wherein the index entry is stored in a file that includes the one or more multimedia content streams.

11. A method as recited in claim 1, wherein the index entry is multiplexed into a data stream that includes the one or more multimedia content streams.

12. A method as recited in claim 1, wherein the one or more criteria for the arbitrary filter and index type comprise receiving input from a user that identifies one or more of the user'"'"'s favorite scenes.

13. For a system capable of processing multimedia content, a computer program product comprising one or more computer readable media carrying computer executable instructions that implement a method of adding an arbitrary index to the multimedia content such that the arbitrary index is capable of coexisting with one or more other arbitrary indexes, the method comprising acts of:

receiving multimedia content comprising one or more samples;

passing at least one sample to one or more multimedia filters for analysis based on one or more criteria that define an index type for each of the one or more multimedia filters;

determining that the at least one sample meets the one or more criteria for an arbitrary filter and index type;

creating an index entry that identifies the at least one sample and the index type of the arbitrary filter, the index entry being adapted in accordance with a common, extensible indexing scheme to coexist and not interfere with one or more other index entries of either a same or different type for the at least one sample; and

storing the index entry for subsequent access.

14. A computer program product as recited in claim 13, wherein the index entry that identifies the at least one sample and index type of the arbitrary filter is created after the content is authored.

15. A computer program product as recited in claim 13, the method further comprising acts of:

passing the at least one sample to a plurality of multimedia filters for analysis based on one or more criteria that define an index type for each of the plurality of multimedia filters;

determining that the at least one sample meets the one or more criteria for multiple filters and index types;

creating a plurality of index entries, each identifying the at least one sample and the index type for one of the multiple filters, the index entries being adapted in accordance with a common, extensible indexing scheme to coexist and not interfere with each other; and

storing the plurality of index entries for subsequent access.

16. A computer program product as recited in claim 13, wherein the one or more criteria for the arbitrary filter and index type comprise receiving input that correlates one or more external events to the at least one sample.

17. A computer program product as recited in claim 16, wherein a first multimedia content filter and a second multimedia content filter are in parallel and are passed at least two samples from two different multimedia streams simultaneously, and wherein the second multimedia filter has access to analysis performed by the first multimedia filter, the method further comprising the second multimedia filter using the analysis performed by the first multimedia filter in analyzing one of the at least two samples so that the second multimedia filter need not duplicate one or more portions of the analysis performed by the first multimedia filter.

18. A computer program product as recited in claim 17, wherein the at least two samples comprise an audio sample from an audio stream and a video sample from a video stream.

19. A computer program product as recited in claim 13, wherein the index entry is stored in a file that is separate from the one or more multimedia content streams.

20. A computer program product as recited in claim 13, wherein the one or more multimedia content streams comprise closed captioning content, and wherein the one or more criteria for the arbitrary filter comprise closed captioning content criteria for identifying a new sentence.

21. A computer program product as recited in claim 13, wherein the one or more multimedia content streams comprise text content, and wherein the one or more criteria for the arbitrary filter comprise text content criteria.

22. A computer program product as recited in claim 13, wherein the system capable of processing multimedia content comprises a personal video recorder.

23. For a system capable of processing multimedia content, a computer program product comprising one or more computer readable media carrying computer executable instructions that implement a method of adding an arbitrary index to the multimedia content such that the arbitrary index is capable of coexisting with one or more other arbitrary indexes, the method comprising steps for:

obtaining one or more multimedia content streams, each comprising one or more samples;

sending at least one sample to one or more multimedia filters for analysis based on one or more criteria that define an index type for each of the one or more multimedia filters;

concluding that the at least one sample meets the one or more criteria for an arbitrary filter and index type;

generating an index entry that identifies the at least one sample and the index type of the arbitrary filter, the index entry being adapted in accordance with a common, extensible indexing scheme to coexist and not interfere with one or more other index entries of either a same or different type for the at least one sample; and

saving the index entry for subsequent access.

24. A computer program product as recited in claim 23, wherein the at least one sample has been indexed previously with an index of the same type, which is kept separate from the generated and saved index entry.

25. A computer program product as recited in claim 23, further comprising acts of:

sending the at least one sample to a plurality of multimedia filters for analysis based on one or more criteria that define an index type for each of the plurality of multimedia filters;

concluding that the at least one sample meets the one or more criteria for multiple filters and index types;

generating a plurality of index entries, each identifying the at least one sample and the index type for one of the multiple filters, the index entries being adapted in accordance with a common, extensible indexing scheme to coexist and not interfere with each other; and

saving the plurality of index entries for subsequent access.

26. A computer program product as recited in claim 25, wherein at least two of the multiple multimedia filters are in series and are passed the at least one sample sequentially.

27. A computer program product as recited in claim 23, wherein the index entry is stored in a temporary memory.

28. A computer program product as recited in claim 23, wherein the one or more criteria for the arbitrary filter and index type comprise receiving input from a user that identifies one or more of the user'"'"'s favorite scenes.

29. A computer program product as recited in claim 23, wherein the one or more multimedia content streams comprise audio content, and wherein the one or more criteria for the arbitrary filter comprise audio content criteria for identifying a new speaker.

30. A computer program product as recited in claim 23, wherein the one or more multimedia content streams comprise video content, and wherein the one or more criteria for the arbitrary filter comprise video content criteria.

31. A computer program product as recited in claim 23, wherein the one or more multimedia content streams comprise image content, and wherein the one or more criteria for the arbitrary filter comprise image content criteria.

32. A computer program product as recited in claim 23, wherein the method further comprises maintaining state information for the one or more multimedia filters to be used in subsequent analysis.

33. A computer program product as recited in claim 23, wherein the index entry identifies the at least one sample indirectly through one or more other index entry within an index entry hierarchy.

34. For a system capable of processing multimedia content, a computer program product comprising one or more computer readable media carrying computer executable instructions that implement a method of interfacing with a plurality of code samples for interacting with an arbitrary index to the multimedia content such that the arbitrary index is capable of coexisting with one or more other arbitrary indexes, the method comprising acts of:

defining a get-index programming interface that retrieves one of one or more indexes for the multimedia content, the get-index programming interface comprising (i) a parameter to identify the one index to retrieve, (ii) a parameter to identify the retrieved one index, and (iii) a return value to indicate whether or not a particular call to the get-index programming interface succeeds;

defining a get-index-from-position programming interface that retrieves one of one or more indexes for the multimedia content based on a position of the one index within the one or more available indexes, the get-index-from-position programming interface comprising (i) a parameter to identify the position of the one index to retrieve, (ii) a parameter to identify the retrieved one index, and (iii) a return value to indicate whether or not a particular call to the get-index-from-position programming interface succeeds; and

defining a get-index-position programming interface that retrieves a position of one of one or more indexes within the one or more indexes for the multimedia content, the get-index-position programming interface comprising (i) a parameter to identify the one index to retrieve, (ii) the position of the one index within the one or more indexes, and (iii) a return value to indicate whether or not a particular call to the get-index-position programming interface succeeds.

35. A computer program product as recited in claim 34, wherein the method further comprises an act of:

defining a get-count programming interface that retrieves a count of the number of indexes available for the multimedia content, the get-count programming interface comprising (i) a parameter to identify the count of the number of indexes available for the multimedia content, and (ii) a return value to indicate whether or not a particular call to the get-count programming interface succeeds.

36. A computer program product as recited in claim 34, wherein the method further comprises an act of:

defining a get-entry-from-position programming interface that retrieves an index entry based on a position of the index entry within an index, the get-entry-from-position programming interface comprising (i) a parameter to identify the position of the index entry within the index, (ii) a parameter to identify a time that corresponds to the index entry, (iii) a parameter for data corresponding to the index entry, (iv) a parameter that specifies on input a data size available for data corresponding to the index entry, and on output specifies an actual data size for data corresponding to the index entry, and (v) a return value to indicate whether or not a particular call to the get-entry-from-position programming interface succeeds.

37. A computer program product as recited in claim 34, wherein the method further comprises an act of:

defining a get-entry-from-time programming interface that retrieves an index entry based on a time to search for the index entry within the multimedia content, the get-entry-from-time programming interface comprising (i) a parameter to identify the time to search for the index entry within the multimedia content, (ii) a parameter to identify a position of the index entry within the index, (iii) a parameter for data corresponding to the index entry, (iv) a parameter that specifies on input a data size available for data corresponding to the index entry, and on output specifies an actual data size for data corresponding to the index entry, and (v) a return value to indicate whether or not a particular call to the get-entry-from-time programming interface succeeds.

38. A computer program product as recited in claim 34, wherein the get-entry-from-time programming interface further comprises an optional parameter to identify a search direction relative to the time to search indicating whether the index entry should appear before, at, after, or as close as possible to the time to search.

39. A computer program product as recited in claim 34, wherein the method further comprises an act of:

defining a get-count programming interface that retrieves a count of index entries within an index, the get-count programming interface comprising (i) a parameter to identify the count of the index entries within the index, and (ii) a return value to indicate whether or not a particular call to the get-count programming interface that retrieves the count of index entries within an index succeeds.

40. A computer program product as recited in claim 34, wherein the method further comprises an act of:

defining a get-descriptor programming interface that retrieves a descriptor of an index, the get-descriptor programming interface comprising (i) a parameter to identify the descriptor of the index, (ii) a parameter to identify on input a data size available for the descriptor, and on output to identify an actual data size of the descriptor, and (iii) a return value to indicate whether or not a particular call to the get-descriptor programming interface succeeds.