Real-time annotation and enrichment of captured video

• US 8,903,798 B2
• Filed: 05/28/2010
• Issued: 12/02/2014
• Est. Priority Date: 05/28/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: El-Saban, Motaz Ahmed, Wang, Xin-Jing, Sayed, May Abdelreheem
• Primary Examiner(s): Mahmoudi, Tony
• Assistant Examiner(s): NGUYEN, MERILYN P
• Application Number:
  US12/790,761
• Publication Number:
  US 20110295851A1
• Time in Patent Office: 1,649 Days
• Field of Search: 707/708, 707/728, 707/729, 707/748, 707/771, 707/913
• US Class Current: 707/708
• CPC Class Codes: G06F 16/748 Hypervideo linking data to content, e.g. by linking an URL to a video object in the context of video distribution systems H04N21/858
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An annotation suggestion platform may comprise a client and a server, where the client captures a media object and sends the captured object to the server, and the server provides a list of suggested annotations for a user to associate with the captured media object. The user may then select which of the suggested metadata is to be associated or stored with the captured media. In this way, a user may more easily associate metadata with a media object, facilitating the media object'"'"'s search and retrieval. The server may also provide web page links related to the captured media object. A user interface for the annotation suggestion platform is also described herein, as are optimizations including indexing and tag propagation.

Claims

1. A method comprising:

identifying, by a computing device, candidate n-grams from metadata associated with a plurality of media objects, the candidate n-grams comprising metadata candidates for annotating a captured media object;

searching for the candidate n-grams in a dictionary;

discarding one or more of the candidate n-grams that are not found in the dictionary;

calculating confidence scores for remaining candidate n-grams following the discarding;

returning one or more of the remaining candidate n-grams as annotation suggestions for a captured media object, each of the one or more of the remaining candidate n-grams having a confidence score that satisfies a predetermined confidence criterion; and

providing, by the computing device, the annotation suggestions to a client device to enable a user selection of at least one of the annotation suggestions as an annotation of the captured media object.

2. The method of claim 1, further comprising:

extracting features from the captured media object;

searching for the features against a previously prepared index of mapping features to the plurality of media objects; and

retrieving the plurality of media objects from a group of media objects in response to determining that the plurality of media objects have features mapped in the previously prepared index that are similar to the features extracted from the captured media object.

3. The method of claim 1, wherein the identifying includes:

determining whether a number of candidate n-grams is less than a predetermined threshold;

generating a new n-gram in response to determining that the number of the candidate n-grams is less than the predetermined threshold, the generating of the new n-gram being performed via one of a machine learning algorithm or a data mining algorithm;

applying a noise filter to the new n-gram;

discarding the new n-gram in response to determining that the new n-gram does not satisfy a noise criteria; and

storing the new n-gram as one of the candidate n-grams in response to determining that the new n-gram satisfies the noise criteria.

4. The method of claim 1, wherein the captured media object is a video object comprising one or more frames, further comprising performing tag propagation, the tag propagation including:

extracting a metadata set for a selected frame in the video object;

selecting a predetermined number of preceding frames in the video object;

retrieving a corresponding metadata set for each of the preceding frames from a buffer;

computing a corresponding similarity score for each of the preceding frames with respect to the selected frame;

propagating at least a portion of metadata in the corresponding metadata set of each preceding frame with the corresponding similarity score that is within a predetermined threshold to the metadata set of the selected frame; and

storing in the buffer the metadata set of the selected frame.

5. The method of claim 4, the method further comprising:

computing a first input score for a preceding frame based on a first similarity metric; and

computing a second input score for the preceding frame based on a second similarity metric,

wherein the similarity score for the preceding frame is computed from at least the first and second input scores.

6. The method of claim 4, wherein each similarity score is weighted by a distance of a corresponding preceding frame to the selected frame.

7. The method of claim 6, the method further comprising:

computing a confidence score for the selected frame based on stored metadata set of the selected frame; and

storing the confidence score for the selected frame to enable tag propagation for subsequent frames.

8. The method of claim 1, wherein the annotation suggestions include a hyperlink to a web page.

9. A method to propagate metadata within video objects, the method comprising:

extracting a metadata set for a selected frame in a video object;

selecting a predetermined number of preceding frames in the video object;

retrieving a metadata set for each of the preceding frames from a buffer;

computing a respective similarity score for each of the preceding frames with respect to the selected frame;

computing a respective confidence score for each metadata set belonging to the preceding frames;

computing a corresponding final score for each preceding frame based on a corresponding similarity score and a corresponding confidence score;

propagating at least a portion of metadata in a corresponding metadata set of each preceding frame with the corresponding final score that is within a predetermined threshold to the metadata set of the selected frame; and

storing in the buffer the metadata set of the selected frame.

10. The method of claim 9, the method further comprising:

computing a first input score for a preceding frame based on a first similarity metric; and

computing a second input score for the preceding frame based on a second similarity metric,

wherein the similarity score is computed from at least the first and second input scores.

11. The method of claim 9, wherein a confidence score indicates a degree of relevance of the corresponding metadata set to a corresponding preceding frame.

12. The method of claim 11, wherein each similarity score is weighted by a distance of a corresponding preceding frame to the selected frame.

13. The method of claim 9, wherein the metadata set of the selected frame include at least one of a text description or a hyperlink to a web page.

14. A system comprising:

a processor; and

a computer readable memory containing computer executable-instructions, the computer-executable instructions configured to be executed by the processor to perform operations including;

extracting a metadata set for a selected frame in a video object;

selecting a predetermined number of preceding frames in the video object;

retrieving a metadata set for each of the preceding frames from a buffer;

computing a corresponding confidence score for each metadata set belonging to the preceding frames;

computing a corresponding similarity score for each of the preceding frames with respect to the selected frame;

computing a corresponding final score for each of the preceding frames based on a corresponding similarity score and a corresponding confidence score;

propagating at least a portion of metadata in a corresponding metadata set of each preceding frame with the corresponding final score that is within a predetermined threshold to the metadata set of the selected frame; and

storing in the buffer the metadata set of the selected frame.

15. The system of claim 14, wherein the computing a corresponding similarity score for a preceding frame comprises:

computing a first input score for the preceding frame based on a first similarity metric;

computing a second input score for the preceding frame based on a second similarity metric; and

computing the corresponding similarity score from at least the first and second input scores.

16. The system of claim 14, wherein each similarity score is weighted by a distance of a corresponding preceding frame to the selected frame.

17. The system of claim 14, wherein a confidence score indicates a degree of relevance of the corresponding metadata set to a corresponding preceding frame.

18. The system of claim 14, wherein the metadata set of the selected frame include at least one of a text description or a hyperlink to a web page.