Structured labeling to facilitate concept evolution in machine learning

• US 10,318,572 B2
• Filed: 02/10/2014
• Issued: 06/11/2019
• Est. Priority Date: 02/10/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kulesza, Todd, Charles, Denis, Caruana, Rich, Amershi, Saleema Amin, Fisher, Danyel Aharon
• Primary Examiner(s): Pitaro, Ryan F
• Assistant Examiner(s): Lobo, Carl P
• Application Number:
  US14/176,990
• Publication Number:
  US 20150227531A1
• Time in Patent Office: 1,947 Days
• CPC Class Codes: G06F 16/287 Visualization; Browsing
• CPC Class Codes: G06F 16/358 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0486 Drag-and-drop

Abstract

A system, method, and media are provided for generating a structured labeling graphical user interface. The user interface receives user input that associates multimedia content with categories. The user input may include user-supplied tags that further define the category for the multimedia content. The user-supplied tags are rendered proximate to the categories. In turn, a database logs user events to store, among other things, the categories, the user-supplied tags, time associated with completing the user-supplied tags, and time for associating multimedia content with the categories or tags.

Claims

1. A computer-implemented method for multimedia content labeling, the method comprising:

providing a plurality of multimedia content items for presentation at a graphical user interface (“

GUI”

), the plurality of multimedia content items comprising at least one of web pages or images;

providing a concept prompt for presentation at the GUI, the concept prompt identifying a concept and prompting a user to indicate whether the plurality of multimedia content items are associated with the concept;

providing a plurality of categories for presentation at the GUI, the plurality of categories having labels indicating user responses to the concept prompt;

receiving an indication of a first user input, the first user input indicating an association between a first multimedia content item of the plurality of multimedia content items and a first category of the plurality of categories, the first category having a first label indicating a user response to the concept prompt for the first multimedia content item;

forming a first group that is presented within the first category, the first group including the first multimedia content item;

receiving a user-supplied tag that describes the first group and providing the user-supplied tag for presentation in association with the first group;

determining that a second multimedia content item is uncategorized and, based on the determination that the second multimedia content item is uncategorized and on at least one of an item-to-group similarity or an item-to-item similarity, generating a suggestion that the second multimedia content item be associated with the first group within the first category, wherein the at least one of the item-to-group similarity or the item-to-item similarity is determined based on at least one of a shortest link, a cosine similarity metric, or a term frequency-inverse document frequency;

providing the suggestion for presentation at the GUI; and

training a machine-learning algorithm with data indicating the association between the first multimedia content item and the first category to provide an improved machine-learning algorithm.

2. The method of claim 1, wherein the labels indicating the user responses to the concept prompt comprise yes, no, and could be.

3. The method of claim 1, wherein visual indicators represent a number of multimedia content items associated with the plurality of categories and are displayed proximate to the plurality of categories.

4. The method of claim 3, wherein the visual indicators are thumbnails of the plurality of multimedia content items or icons.

5. The method of claim 3, wherein the visual indicators comprise numerical values indicating a count of the number of multimedia content items associated with the plurality of categories.

6. The method of claim 1, further comprising:

providing a viewable region in the GUI with multimedia content similar to multimedia content associated with the first category.

7. The method of claim 1, wherein the first user input comprises dragging and dropping a visual representation of the first multimedia content item onto or near at least one of a visual representation of the first category or a visual representation of the user-supplied tag associated with the first group within the first category.

8. The method of claim 1, wherein the first user input comprises at least one of removing, combining, or adding user-supplied tags to the GUI.

9. The method of claim 1, further comprising:

receiving an indication of a second user input, the second user input indicating a command to merge the first group with a second group of multimedia content items; and

in response to receiving the indication of the second user input, merging the first group with the second group.

10. The method of claim 1, further comprising:

storing a plurality of user input events, the plurality of categories, and the association between the first multimedia content item and the first category in a database, wherein storing the plurality of categories in the database comprises storing a number of multimedia content items associated with each of the plurality of categories and a number of multimedia content items associated with the user-supplied tag.

11. The method of claim 10, wherein storing the plurality of user input events in the database includes storing an amount of time that occurs between each of the plurality of user input events.

12. The method of claim 10, wherein the method further comprises:

generating a summary, the summary comprising (1) a number of revisions made to user-supplied tags and (2) a number of move, combine, add, or delete inputs received for each of a plurality of groups within the plurality of categories; and

providing the summary for output.

13. One or more computer storage devices having computer-executable instructions embodied thereon that, when executed, perform a method of generating a structured labeling graphical user interface (“

GUI”

), the method comprising;

providing a concept prompt for presentation at a GUI, the concept prompt identifying a concept and prompting a user to indicate whether multimedia content items are associated with the concept, the GUI having at least two portions, a first portion for displaying the multimedia content items and a second portion for assigning the multimedia content items to one of at least two categories, the at least two categories having labels indicating user responses to the concept prompt, and each of the at least two categories including a plurality of groups that are presented within the respective category;

based on a plurality of inputs provided by the user, associating each of the multimedia content items displayed in the first portion with a group of the plurality of groups included within one of the at least two categories in the second portion of the GUI;

analyzing the multimedia content items associated with a first group of the plurality of groups, wherein analyzing the multimedia content items comprises determining search terms associated with the multimedia content items;

based on the analysis, automatically generating a summary for the first group, the summary comprising at least one of the search terms as a suggested description for the multimedia content items included in the first group;

providing the summary for output;

based on at least one of an item-to-group similarity or an item-to-item similarity, providing a suggestion to associate an uncategorized multimedia content item with a particular group, wherein the at least one of the item-to-group similarity or the item-to-item similarity is determined based on at least one of a shortest link, a cosine similarity metric, or a term frequency-inverse document frequency; and

training a machine-learning algorithm with data indicating the associations between the multimedia content items and the plurality of groups.

14. The computer storage devices of claim 13, further comprising:

generating a third portion of the GUI having additional uncategorized multimedia content items that are similar to each other.

15. The computer storage devices of claim 13, wherein the at least one of the item-to-group similarity or the item-to-item similarity comprises a similarity between the uncategorized multimedia content item and a categorized multimedia content item that is associated with the particular group.

19. The computer storage devices of claim 13, wherein determining the search terms associated with the multimedia content items comprises:

accessing a search query log comprising search queries that returned the multimedia content items; and

extracting the search terms from the search queries.

20. The computer storage devices of claim 13, wherein analyzing the multimedia content items associated with the first group comprises determining the most frequently occurring terms associated with the multimedia content items.

16. A computer system comprising:

one or more processors; and

computer storage memory having stored thereon computer-executable instructions that, when executed by the one or more processors, implement a method comprising;

providing a plurality of multimedia content items for presentation at a graphical user interface (“

GUI”

);

providing a concept prompt for presentation at the GUI, the concept prompt identifying a concept and prompting a user to indicate whether the plurality of multimedia content items are associated with the concept;

providing a plurality of categories for presentation at the GUI, the plurality of categories having labels indicating user responses to the concept prompt;

receiving an indication of a first user input, the first user input indicating an association between a first multimedia content item of the plurality of multimedia content items and a first group that is presented within a first category of the plurality of categories, the first category having a first label indicating a user response to the concept prompt for the first multimedia content item;

providing a user-supplied tag for presentation in association with the first group;

based on at least one of an item-to-group similarity or an item-to-item similarity, generating a suggestion that a second multimedia content item be associated with the first group, wherein the at least one of the item-to-group similarity or the item-to-item similarity is determined based on at least one of a shortest link, a cosine similarity metric, or a term frequency-inverse document frequency;

providing the suggestion for presentation at the GUI; and

training a machine-learning algorithm with data indicating the association between the first multimedia content item and the first category.

17. The system of claim 16, wherein the method further comprises:

receiving an indication of a second user input, the second user input indicating a command to move the first group from the first category to a second category of the plurality of categories, the first category comprising a maybe category and the second category comprising a yes category; and

in response to the indication of the second user input, moving the first group from the first category to the second category.

18. The system of claim 16, wherein the method further comprises:

generating a summary, the summary comprising a number of user-provided revisions received for user-supplied tags and a number of user-provided commands associated with a plurality of groups included within the plurality of categories, the user-provided commands comprising move, merge, add, and delete commands; and

providing the summary for output.