Identification of segments within audio, video, and multimedia items

• US 9,442,933 B2
• Filed: 12/24/2008
• Issued: 09/13/2016
• Est. Priority Date: 12/24/2008
• Status: Active Grant

• Current Assignee: TiVo Corporation (Adeia Inc.)
• Original Assignee: Comcast Interactive Media LLC (Comcast Corporation)
• Inventors: Tzoukermann, Evelyne, Chipman, Leslie Eugene, Davis, Anthony R., Houghton, David F., Farrell, Ryan M., Zhou, Hongzhong, Jojic, Oliver, Kronrod, Vladimir, Shevade, Bageshree, Ambwani, Geetu
• Primary Examiner(s): DANG, HUNG Q
• Application Number:
  US12/343,779
• Publication Number:
  US 20100158470A1
• Time in Patent Office: 2,820 Days
• Field of Search: 386239-262, 386278-290
• US Class Current: 1/1
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

The invention pertains to methods, systems, and apparatus for identifying segments within a media item, the media segment including at least one of audio content and video content, comprising segmenting the media item into a plurality of segments as a function of subject matter, storing data identifying each segment and its subject matter, and organizing each segment within an ontology based on its subject matter.

Claims

1. A method, comprising:

analyzing, to determine one or more keywords, descriptive data of a media item;

analyzing, by a computing device, the one or more keywords and selecting, based on the analyzing of the one or more keywords, a knowledge domain for the media item from a plurality of different knowledge domains;

identifying, based on the knowledge domain, a plurality of attribute fields associated with the knowledge domain;

identifying, from different combinations of media analysis technologies that correspond to the plurality of different knowledge domains and that are usable to analyze media, a combination of media analysis technologies corresponding to the knowledge domain;

analyzing, after identifying the combination of media analysis technologies, the media item using the combination of media analysis technologies to determine values for the plurality of attribute fields; and

segmenting the media item into a plurality of segments as a function of the values for the plurality of attribute fields by at least determining beginning and ending boundaries for the plurality of segments as a function of the values for the plurality of attribute fields.

2. The method of claim 1, wherein the media item comprises at least one of audio content or video content, wherein the combination of media analysis technologies comprises a first media analysis technology, and wherein analyzing the media item using the combination of media analysis technologies to determine the values for the plurality of attribute fields comprises:

analyzing, using the first media analysis technology, the at least one of the audio content or the video content to determine one or more of the values for the plurality of attribute fields, wherein the plurality of attribute fields and the values for the plurality of attribute fields form a plurality of attribute/value pairs; and

wherein the method further comprises;

associating the plurality of attribute/value pairs with at least one of the plurality of segments.

3. The method of claim 2 further comprising obtaining, from an information source different from a source of the media item, information about the media item;

and

wherein segmenting the media item into the plurality of segments is performed as a function of the values for the plurality of attribute fields and the information about the media item.

4. The method of claim 1 further comprising:

classifying each of the plurality of segments within an ontology, resulting in one or more classifications of each segment; and

storing information identifying the plurality of segments to allow the plurality of segments to be found in future searches and storing information identifying the one or more classifications.

5. The method of claim 1 further comprising:

storing information identifying the plurality of segments to allow the plurality of segments to be found in future searches; and

performing, responsive to a search query, a search for segments based on the information identifying the plurality of segments.

6. The method of claim 1 wherein the media item comprises at least one of audio content or video content;

wherein the combination of media analysis technologies comprises at least one of a closed captioning analytics technology, speech recognition technology, audio analytics technology for analyzing non-speech aspects of audio of the audio content, optical character recognition technology, video analytics technology, or metadata analytics technology; and

wherein analyzing the media item using the combination of media analysis technologies to determine the values for the plurality of attribute fields comprises at least one of analyzing a closed captioning stream of the media item using the closed captioning analytics technology, analyzing the audio content of the media item for non-speech contextual information using the audio analytics technology, performing speech recognition on the audio content of the media item using the speech recognition technology, performing optical character recognition on the video content of the media item using the optical character recognition technology, performing video analytics on the video content of the media item using the video analytics technology, or analyzing metadata of the media item using the metadata analytics technology.

7. The method of claim 1 wherein the media item comprises multimedia assets of audio or video, and wherein the plurality of segments are sub-assets of the multimedia assets.

8. The method of claim 1 wherein the media item comprises television programming.

9. The method of claim 1, wherein the plurality of attribute fields is unique to the knowledge domain.

10. The method of claim 1, wherein the plurality of attribute fields comprises a set of knowledge domain specific media segmentation parameters that identify features to look for when segmenting media of the knowledge domain.

11. The method of claim 1, further comprising:

storing the values for the plurality of attribute fields to a database; and

storing the beginning and the ending boundaries for the plurality of segments to the database.

12. A method, comprising:

storing a first set of attribute fields defined for a first knowledge domain;

storing a second set of attribute fields defined for a second knowledge domain different from the first knowledge domain;

storing data describing a first gathering process that is specific to the first knowledge domain and that identifies a first combination of media analysis technologies to use when segmenting media of the first knowledge domain;

storing, by a computing device, data describing a second gathering process that is specific to the second knowledge domain and that identifies a second combination of media analysis technologies to use when segmenting media of the second knowledge domain, the second combination of media analysis technologies being different from the first combination of media analysis technologies;

analyzing, to determine one or more keywords, descriptive data of a media item;

analyzing the one or more keywords and selecting, based on the analysis of the one or more keywords, the first knowledge domain;

identifying, after the first knowledge domain is selected, the first set of attribute fields;

retrieving the data describing the first gathering process;

analyzing, after retrieving the data describing the first gathering process, the media item using the first combination of media analysis technologies to determine values for the first set of attribute fields; and

segmenting the media item into a plurality of segments as a function of the first gathering process and the values for the first set of attribute fields by at least determining beginning and ending boundaries for the plurality of segments as a function of the values for the first set of attribute fields.

13. The method of claim 12 further comprising:

analyzing, to determine one or more second keywords, descriptive data of a second media item;

analyzing the one or more second keywords and selecting, based on the analysis of the one or more second keywords, the second knowledge domain;

identifying, after the second knowledge domain is selected, the second set of attribute fields;

retrieving the data describing the second gathering process;

analyzing, after retrieving the data describing the second gathering process, the second media item using the second combination of media analysis technologies to determine values for the second set of attribute fields; and

segmenting the second media item into a second plurality of segments as a function of the second gathering process and the values for the second set of attribute fields by at least determining beginning and ending boundaries for the second plurality of segments as a function of the values for the second set of attribute fields.

14. The method of claim 13 wherein the first knowledge domain is a first type of sport, and the second knowledge domain is a second type of sport.

15. The method of claim 14 wherein the first set of attribute fields comprises at least one of the following:

a team on offense attribute field, a team on defense attribute field, a game time attribute field, a down number attribute field, a key offensive players attribute field, a key defensive player attribute field, a type of play attribute field, or a yards gained attribute field; and

wherein the second set of attribute fields comprises at least one of the following;

an inning number attribute field, a batter attribute field, or a result of at-bat attribute field.

16. The method of claim 12 wherein the media item comprises at least one of audio content or video content;

wherein the first combination of media analysis technologies comprises at least one of a closed captioning analytics technology, speech recognition technology, audio analytics technology for analyzing non-speech aspects of audio of the audio content, optical character recognition technology, video analytics technology, or metadata analytics technology; and

wherein analyzing the media item using the first combination of media analysis technologies to determine the values for the first set of attribute fields comprises at least one of analyzing a closed captioning stream of the media item using the closed captioning analytics technology, analyzing the audio content for non-speech contextual information using the audio analytics technology, performing speech recognition on the audio content using speech recognition technology, performing optical character recognition on the video content using the optical character recognition technology, performing video analytics on the video content using the video analytics technology, or analyzing metadata of the media item using the metadata analytics technology.

17. The method of claim 12, wherein the first combination of media analysis technologies comprises speech recognition technology, video analytics technology and metadata analytics technology, and

wherein the second combination of media analysis technologies comprises audio analytics technology for analyzing non-speech aspects of audio of the media item and optical character recognition technology.

18. The method of claim 12, further comprising:

storing the values for the first set of attribute fields to a database; and

storing the beginning and the ending boundaries for the plurality of segments to the database.

19. A method, comprising:

analyzing, to determine one or more keywords, descriptive data of a media item, the media item comprising video content;

analyzing, by a computing device, the one or more keywords and selecting, based on the analyzing of the one or more keywords, a knowledge domain for the media item from a plurality of different knowledge domains;

identifying, based on the knowledge domain, a plurality of attribute fields associated with the knowledge domain;

selecting, from different pre-defined combinations of media analysis technologies corresponding to the plurality of different knowledge domains, a pre-defined combination of media analysis technologies that corresponds to the knowledge domain and that comprises a first media analysis technology for analyzing the video content;

analyzing, after selecting the pre-defined combination of media analysis technologies, the media item using the pre-defined combination of media analysis technologies to determine values for the plurality of attribute fields by at least analyzing the video content of the media item using the first media analysis technology to determine one or more first values of the values for the plurality of attribute fields; and

segmenting the media item into a plurality of segments as a function of the values for the plurality of attribute fields by at least determining beginning and ending boundaries for the plurality of segments as a function of the values for the plurality of attribute fields.

20. The method of claim 19, wherein the media item comprises audio content;

wherein the pre-defined combination of media analysis technologies comprises a second media analysis technology for analyzing the audio content; and

wherein analyzing the media item using the pre-defined combination of media analysis technologies to determine the values for the plurality of attribute fields comprises analyzing the audio content using the second media analysis technology to determine one or more second values of the values for the plurality of attribute fields.

21. The method of claim 20, wherein the second media analysis technology is software for analyzing the audio content to determine an identity of a person speaking within the audio content, and wherein the one or more second values comprises the identity of the person speaking within the audio content.

22. The method of claim 20, wherein the second media analysis technology is software for analyzing the audio content to determine an occurrence of a particular type of sound within the audio content, and wherein the one or more second values comprises an indication of the occurrence of the particular type of sound.

23. The method of claim 19, wherein the media item comprises metadata;

wherein the pre-defined combination of media analysis technologies comprises a second media analysis technology for analyzing the metadata; and

wherein analyzing the media item using the pre-defined combination of media analysis technologies to determine the values for the plurality of attribute fields comprises analyzing the metadata using the second media analysis technology to determine one or more second values of the values for the plurality of attribute fields.