Enriching online videos by content detection, searching, and information aggregation

• US 9,443,147 B2
• Filed: 04/26/2010
• Issued: 09/13/2016
• Est. Priority Date: 04/26/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Mei, Tao, Hua, Xian-Sheng, Li, Shipeng
• Primary Examiner(s): Cao, Phuong Thao
• Application Number:
  US12/767,114
• Publication Number:
  US 20110264700A1
• Time in Patent Office: 2,332 Days
• Field of Search: 707/780, 707/772, 707/914, 707/E17.02, 707/999.107
• US Class Current: 1/1
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/735 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content

Abstract

Many internet users consume content through online videos. For example, users may view movies, television shows, music videos, and/or homemade videos. It may be advantageous to provide additional information to users consuming the online videos. Unfortunately, many current techniques may be unable to provide additional information relevant to the online videos from outside sources. Accordingly, one or more systems and/or techniques for determining a set of additional information relevant to an online video are disclosed herein. In particular, visual, textual, audio, and/or other features may be extracted from an online video (e.g., original content of the online video and/or embedded advertisements). Using the extracted features, additional information (e.g., images, advertisements, etc.) may be determined based upon matching the extracted features with content of a database. The additional information may be presented to a user consuming the online video.

Claims

1. A method of identifying digital content relevant to an online video, comprising:

identifying, by one or more processing units, promotional content embedded within the online video, the promotional content being generated by a first content producer;

extracting, by the one or more processing units, a feature from the promotional content embedded in the online video, the feature comprising at least one of a textual feature, a visual feature, or an audio feature;

searching, by the one or more processing units, a database to identify additional information based upon the feature, the additional information being generated by a second content producer different than the first content producer;

identifying, by the one or more processing units, candidates from the additional information, the candidates including at least two of a textual feature, a visual feature, or an audio feature;

aggregating, by the one or more processing units, the candidates into an aggregate video; and

outputting, by the one or more processing units, the aggregate video for presentation.

2. The method of claim 1, wherein outputting the aggregate video for presentation comprises:

presenting the aggregate video and the promotional content sequentially.

3. The method of claim 1, the visual feature comprising a color histogram.

4. The method of claim 1, the extracting the feature comprising at least one of:

extracting the textual feature using an OCR text recognition technique;

or

extracting the visual feature using a scale-invariant feature transformation.

5. The method of claim 1, the searching the database to identify the additional information comprising:

executing a multi modal relevance matching algorithm against the database using the feature to determine the additional information.

6. The method of claim 1, the aggregate video being formatted to be embedded in the online video.

7. The method of claim 1, the extracting the feature comprising:

parsing the promotional content embedded in the online video into one or more segments, each of the one or more segments comprising a sequence of frames;

for respective segments, extracting one or more keyframes; and

extracting the feature from the one or more keyframes of respective segments.

8. The method of claim 7, the extracting one or more keyframes comprising at least one of:

selecting a middle frame as a keyframe;

or

selecting a frame having a video quality above a threshold as the keyframe.

9. The method of claim 1, wherein the aggregate video comprises advertising information associated with the promotional content embedded within the online video.

10. A system for identifying digital content relevant to an online video, comprising:

one or more processing units; and

a memory storing;

a parsing component configured to, when executed by the one or more processing units;

select a frame of an online video as a keyframe based upon the frame having a video quality above a threshold;

a feature extraction component configured to, when executed by the one or more processing units;

identify promotional content embedded within the keyframe, the promotional content being generated by a first content producer; and

extract a feature from at least the promotional content embedded within the keyframe, the feature comprising at least one of a textual feature, a visual feature, or an audio feature;

an information extraction component configured to, when executed by the one or more processing units;

search a database to identify additional information based upon the feature, the additional information being generated by a second content provider different than the first content producer;

identify candidates from the additional information determined to be associated with the feature extracted from the promotional content, the candidates including at least two of a textual feature, a visual feature, or an audio feature; and

in response to identifying the candidates, automatically aggregate the candidates into an aggregate video; and

a presentation component configured to, when executed by the one or more processing units;

present the aggregate video.

11. The system of claim 10, the presentation component further configured to:

present the online video sequentially with the aggregate video.

12. The system of claim 11, the aggregate video being different than the promotional content.

13. The system of claim 10, the parsing component further configured to:

parse the online video into one or more segments, a segment comprising a sequence of frames; and

for respective segments, extract one or more keyframes.

14. The system of claim 10, the information extraction component further configured to:

execute a multi modal relevance matching algorithm against the database using the feature to determine the additional information.

15. The system of claim 14, the multimodal relevance matching algorithm further configured to:

perform a text based search algorithm upon the database using the textual feature to determine a first list of candidate additional information;

perform a visual feature matching algorithm upon the database using the visual feature to determine a second list of candidate additional information; and

perform a linear combination of the first list and second list to generate a set of additional information.

16. A computer-readable storage device comprising processor-executable instructions that, when executed, cause a computing device to perform a method for identifying digital content relevant to an online video, the method comprising:

selecting a frame of the online video as a keyframe based upon the frame having a video quality above a threshold;

identifying promotional content embedded within the keyframe, the promotional content being generated by a first content producer;

extracting a feature from at least the promotional content embedded in the keyframe, the feature comprising at least one of a textual feature, a visual feature, or an audio feature;

searching a database to identify additional information based upon the feature, the additional information being generated by a second content producer different than the first content producer;

identifying candidates from the additional information determined to be associated with the feature extracted from the promotional content, the candidates including at least two of a texture feature, a visual feature, or an audio feature;

in response to identifying the candidates, automatically aggregating the candidates into an aggregate video; and

outputting the aggregate video for presentation.

17. The computer-readable storage device of claim 16, the method comprising:

presenting the online video sequentially with the aggregate video.