Real-time annotation and enrichment of captured video

• US 9,652,444 B2
• Filed: 11/19/2014
• Issued: 05/16/2017
• Est. Priority Date: 05/28/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wang, Xin-Jing, Sayed, May Abdelreheem, El-Saban, Motaz Ahmed
• Primary Examiner(s): NGUYEN, MERILYN P
• Application Number:
  US14/547,941
• Publication Number:
  US 20150082173A1
• Time in Patent Office: 909 Days
• Field of Search: 707708
• CPC Class Codes: G06F 16/748 Hypervideo linking data to content, e.g. by linking an URL to a video object in the context of video distribution systems H04N21/858
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

An annotation suggestion platform may comprise a client and a server, where the client captures a media object and sends the captured object to the server, and the server provides a list of suggested annotations for a user to associate with the captured media object. The user may then select which of the suggested metadata is to be associated or stored with the captured media. In this way, a user may more easily associate metadata with a media object, facilitating the media object'"'"'s search and retrieval. The server may also provide web page links related to the captured media object. Suggested metadata may be viewed, selected, and associated through a user interface. The user interface may additionally provide space for web pages to be navigated and viewed.

Claims

1. A system comprising:

a processor;

a display; and

a computer readable memory containing computer executable instructions, the computer-executable instructions configured to be executed by the processor to perform operations including;

transmitting a captured media object to a remote server;

receiving from the remote server a plurality of suggested descriptions for the captured media object and a first plurality of suggested links to web pages;

displaying the plurality of suggested descriptions for the captured media object in a user metadata selection form, the user metadata selection form including a toggle to allow toggling between the plurality of suggested descriptions and the first plurality of suggested links to web pages;

receiving a first input to select at least one suggested description of the plurality of suggested descriptions from the user metadata selection form, the first input indicating that the at least one suggested description is relevant to the captured media object;

receiving a second input to de-select at least one suggested description of the plurality of suggested descriptions from the user metadata selection form, the second input indicating that the at least one suggested description is irrelevant to the captured media object;

transmitting the first input to the remote server;

transmitting the second input to the remote server;

receiving, from the remote server, a second plurality of links to web pages that are determined to be relevant to the captured media object, the second plurality of links being determined based at least in part on the captured media object, the first input, and the second input; and

displaying the second plurality of links.

2. The system of claim 1, wherein the system further comprises one or more input controls, and the computer executable instructions are further configured for:

displaying a user preference form on a display;

receiving user preference information in response to the user preference form using at least one input control of the one or more input controls; and

transmitting over a cellular subsystem the user preference information to the remote server.

3. The system of claim 1, wherein receiving the first input to select the at least one suggested description of the plurality of suggested descriptions from the user metadata selection form comprises:

receiving user metadata selection input in response to the user metadata selection form; and

transmitting over a cellular subsystem the user metadata selection input.

4. The system of claim 3, wherein the second plurality of links to web pages are further determined, at least in part, on the user metadata selection input.

5. The system of claim 3, wherein the computer executable instructions are further configured for:

identifying selected ones of the plurality of suggested descriptions from the user metadata selection input; and

storing locally on the system the selected ones of the plurality of suggested descriptions and storing associations of the selected ones of the plurality of suggested descriptions with the captured media object.

6. The system of claim 1, wherein the computer executable instructions are further configured for displaying thumbnails of the web pages corresponding to the second plurality of links on the display.

7. The system of claim 6, wherein the computer executable instructions are further configured for:

receiving a user web page selection using at least one input control;

transmitting over the cellular subsystem the user web page selection;

receiving web page content, based on the user web page selection; and

displaying the web page content on the display.

8. The system of claim 1, wherein the plurality of suggested descriptions of the captured media object have a confidence score that satisfied a predetermined confidence criterion.

9. A method comprising:

transmitting, by a computing device, a captured media object to a remote server;

receiving, by the computing device and from the remote server, a plurality of suggested descriptions for the captured media object and a first plurality of suggested links to web pages;

receiving, by the computing device, a first input to select at least one suggested description of the plurality of suggested descriptions from the user metadata selection form, the first input indicating that the at least one suggested description is relevant to the captured media object, the user metadata selection form including a toggle to allow toggling between the plurality of suggested descriptions and the first plurality of suggested links to web pages;

receiving, by the computing device, a second input to de-select at least one suggested description of the plurality of suggested descriptions from the user metadata selection form, the second input indicating that the at least one suggested description is irrelevant to the captured media object;

transmitting, by the computing device, the first input to the remote server;

transmitting, by the computing device, the second input to the remote server;

receiving, by the computing device, a second plurality of links to web pages that are determined to be relevant to the captured media object from the remote server, the second plurality of links to the web pages being determined based at least in part on the captured media object, the first input, and the second input; and

displaying the second plurality of links to the web pages.

10. The method of claim 9, further comprising:

displaying a user preference form on a display;

receiving user preference information in response to the displaying of the user preference form using at least one input control; and

transmitting over a cellular subsystem the user preference information to the remote server.

11. The method of claim 9, further comprising:

displaying the plurality of suggested descriptions in a user metadata selection form on a display;

receiving user metadata selection input in response to the user metadata selection form; and

transmitting over a cellular subsystem the user metadata selection input.

12. The method of claim 11, wherein the second plurality of links to the web pages are further determined based at least on the user metadata selection input.

13. The method of claim 12, further comprising displaying thumbnails of the web pages corresponding to the second plurality of links on the display.

14. The method of claim 13, further comprising:

receiving a user web page selection using at least one input control;

transmitting over the cellular subsystem the user web page selection;

receiving web page content, based on the user web page selection; and

displaying the web page content on the display.

15. The method of claim 11, further comprising:

identifying selected ones of the plurality of suggested descriptions from the user metadata selection input; and

storing locally on the computing device selected ones of the plurality of suggested descriptions corresponding to the selected user metadata input and storing associations of the selected ones of the plurality of suggested descriptions with the captured media object.

16. The method of claim 9, wherein the plurality of suggested descriptions of the captured media object have confidence scores that satisfy a predetermined confidence criterion.

17. An apparatus comprising:

a processor;

a display; and

a computer readable memory having, computer-executable instructions stored thereon which, when executed by a computing device, cause the computing device to perform operations comprising;

transmitting a captured media object to a remote server;

receiving from the remote server a plurality of suggested descriptions for

the captured media object and a first plurality of suggested links to web pages;

displaying the plurality of suggested descriptions for the captured media object in a user metadata selection form, the user metadata selection form including a toggle to allow toggling between the plurality of suggested descriptions and the first plurality of suggested links to web pages;

receiving a first input to select at least one suggested description of the plurality of suggested descriptions from the user metadata selection form, the first input indicating that the at least one suggested description is relevant to the captured media object;

receiving a second input to de-select at least one suggested description of the plurality of suggested descriptions from the user metadata selection form, the second input indicating that the at least one suggested description is irrelevant to the captured media object;

transmitting the first input to the remote server;

transmitting the second input to the remote server;

receiving, from the remote server, a second plurality of links to web pages that are determined to be relevant to the captured media object, the second plurality of links to the web pages being determined based at least in part on the captured media object, the first input, and the second input; and

displaying the second plurality of links to the web pages.

18. The apparatus of claim 17, wherein the operations further comprise:

displaying a user preference form on a display;

receiving user preference information in response to the displaying of the user preference form using at least one input control; and

transmitting over a cellular subsystem the user preference information to the remote server.

19. The apparatus of claim 17, wherein the operations further comprise:

receiving user metadata selection input in response to the user metadata selection form; and

transmitting over a cellular subsystem the user metadata selection input.

20. The apparatus of claim 17, wherein the plurality of suggested descriptions of the captured media object have a confidence score that satisfies a predetermined confidence criterion.