ENRICHING ONLINE VIDEOS BY CONTENT DETECTION, SEARCHING, AND INFORMATION AGGREGATION

• US 20110264700A1
• Filed: 04/26/2010
• Published: 10/27/2011
• Est. Priority Date: 04/26/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mei, Tao, Hua, Xian-Sheng, Li, Shipeng
• Granted Patent: US 9,443,147 B2
• Time in Patent Office: Days
• US Class Current: 707/780
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/735 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06V 20/47 Detecting features for summarising video content
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content

Abstract

Many internet users consume content through online videos. For example, users may view movies, television shows, music videos, and/or homemade videos. It may be advantageous to provide additional information to users consuming the online videos. Unfortunately, many current techniques may be unable to provide additional information relevant to the online videos from outside sources. Accordingly, one or more systems and/or techniques for determining a set of additional information relevant to an online video are disclosed herein. In particular, visual, textual, audio, and/or other features may be extracted from an online video (e.g., original content of the online video and/or embedded advertisements). Using the extracted features, additional information (e.g., images, advertisements, etc.) may be determined based upon matching the extracted features with content of a database. The additional information may be presented to a user consuming the online video.

Claims

1. A method for determining a set of additional information relevant to an online video, comprising:

receiving an online video;

extracting a set of features from the online video, the set of features comprising at least one of a textual feature, a visual feature, and an audio feature; and

determining a set of additional information based upon the set of features.

2. The method of claim 1, comprising:

presenting the set of additional information on a display.

3. The method of claim 2, the presenting comprising at least one of:

presenting the set of additional information as a video;

presenting the set of additional information as an animation;

presenting the set of additional information as an image; and

presenting the set of additional information as text.

4. The method of claim 1, the extracting comprising:

extracting a textual feature using an OCR text recognition technique.

5. The method of claim 1, the extracting comprising:

extracting a visual feature using a scale-invariant feature transformation.

6. The method of claim 1, the visual feature comprising at least one of:

a color histogram;

textures; and

SIFT descriptors.

7. The method of claim 1, the determining comprising:

executing a multimodal relevance matching algorithm against a database using the set of features to determine the set of additional information.

8. A method for determining a set of additional information relevant to an online video, comprising:

receiving an online video;

parsing the online video into one or more segments, a segment comprising a sequence of frames;

for respective segments, extracting one or more keyframes representative of a sequence of frames of a segment;

extracting a set of features from the one or more keyframes of respective segments, the set of features comprising one or more textual features and one or more visual features;

performing a text based search algorithm upon a database using the one or more textual features to determine a first list of candidate additional information;

performing a visual feature matching algorithm upon the database using the one or more visual features to determine a second list of candidate additional information; and

performing a linear combination on the first list and second list to generate a candidate set of additional information.

9. The method of claim 8, the parsing comprising:

determining a boundary between segments based upon a large visual difference between two sequential frames.

10. The method of claim 8, the extracting one or more keyframes comprising at least one of:

selecting a middle frame from the segment as a keyframe; and

selecting a frame having a video quality above a predetermined threshold as the keyframe.

11. The method of claim 8, comprising:

selecting a predetermined number of candidates from within the candidate set of additional information to generate a set of additional information; and

aggregating the set of additional information into a video.

12. The method of claim 11, comprising:

presenting the video to a user engaged with the online video.

13. The method of claim 12, comprising:

presenting the video within a web browser hosting the online video.

14. The method of claim 8, comprising:

designating a first set of segments as promotional content; and

designating a second set of segments program content.

15. A system for determining a set of additional information relevant to an online video, comprising:

a parsing component configured to;

parse an online video into one or more segments, a segment comprising a sequence of frames; and

for respective segments, extract one or more keyframes representative of a sequence of frames of a segment;

a feature extraction component configured to;

extract a set of features from the one or more keyframes of respective segments, the set of features comprising at least one of a textual feature, a visual feature, and an audio feature; and

an information extraction component configured to;

execute a multimodal relevance matching algorithm against a database using the set of features to determine the set of additional information.

16. The system of claim 15, the information extraction component configured to:

aggregate the set of additional information into a video.

17. The system of claim 16, comprising:

a presentation component configured to;

present the video to a user consuming the online video.

18. The method of claim 15, comprising:

a presentation component configured to;

present the set of additional information.

19. The system of claim 15, the feature extraction component comprising:

a text recognition component configured to extract one or more textual features using an OCR text recognition technique;

a visual recognition component configured to extract one or more visual features using a scale-invariant feature transformation; and

an audio recognition component configured to extract one or more audio features.

20. The system of claim 15, the multimodal relevance matching algorithm configured to:

perform a text based search algorithm upon the database using the one or more textual features to determine a first list of candidate additional information;

perform a visual feature matching algorithm upon the database using the one or more visual features to determine a second list of candidate additional information; and

perform a linear combination of the first list and second list to generate the set of additional information.