User interface for context labeling of multimedia items

• US 10,222,942 B1
• Filed: 01/20/2016
• Issued: 03/05/2019
• Est. Priority Date: 01/22/2015
• Status: Active Grant

• Current Assignee: Clarifai, Inc.
• Original Assignee: Clarifai, Inc.
• Inventors: Zeiler, Matthew D., Berenzweig, Adam L.
• Primary Examiner(s): Vu, Thanh T
• Application Number:
  US15/002,248
• Time in Patent Office: 1,140 Days
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06F 7/08 Sorting, i.e. grouping record carriers in numerical or other ordered sequence according to the classification of at least some of the information they carry by merging two or more sets of carriers in ordered sequence G06F7/16
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

Context information may be associated with multimedia items. One or more multimedia items may be obtained from a repository, the multimedia items(s) including one or more of an image, a video, audio, a text file, combinations thereof, and/or other considerations. Predicted context information may be associated with individual ones of the multimedia items. Labels and/or other context information associated with multimedia item(s) may be stored as metadata associated with the multimedia item(s). A user interface may be configured to display one or more of the obtained multimedia items and the predicted context information associated with the one or more obtained multimedia items. Entry and/or selection may be obtained from one or more users of an addition, removal, correction, and/or confirmation of the predicted context information for individual ones of the multimedia items and/or groups of multimedia items displayed in the user interfaces.

Claims

1. A system for facilitating training of a neural network to associate context information with multimedia items, the system comprising:

a computer system comprising one or more processors programmed with computer program instructions that, when executed, cause the computer system to;

provide multimedia items to a neural network to cause the neural network to predict one or more labels for the multimedia items, the neural network predicting a label as a corresponding label for first and second multimedia items of the multimedia items;

assign the first and second multimedia items to a group based on the neural network predicting the label as a corresponding label for the first and second multimedia items,

generate, based on the predicted label, a task related to the predicted label, the task soliciting a user to indicate one or more of the multimedia items that are relevant to the predicted label;

cause the first multimedia item, the second multimedia item, and the task to be presented together on a user interface at a same time based on the assignment of the first and second multimedia items to the group;

obtain, via the user interface, a user response to the task, the user response comprising (i) a first user indication of the first multimedia item as being relevant to the predicted label and (ii) a second user indication of the second multimedia item as being relevant or not relevant to the predicted label; and

provide, to the neural network, the first and second user indications to cause the neural network to be updated based on the first and second user indications.

2. The system of claim 1, wherein the computer system is caused to:

provide, to one or more other neural networks, the first and second user indications to cause the one or more other neural networks to be updated based on the first and second user indications.

3. The system of claim 1, wherein the computer system is caused to:

determine a confidence score associated with the prediction of the label for the first multimedia item,

wherein the task is presented on the user interface prior to presenting one or more other tasks based on the confidence score being lower than other confidence scores associated with other predictions of labels for other multimedia items to which the one or more other tasks are related.

4. The system of claim 1, wherein each of the multimedia items comprising at least one of an image, an audio, or a video.

5. The system of claim 1, wherein the computer system is caused to:

obtain the user response to the task from a user;

obtain one or more other user responses to the task from one or more other users; and

assign the task to one or more additional users based on the user response and the one or more other user responses collectively not satisfying a consensus threshold.

6. The system of claim 5, wherein the computer system is caused to:

obtain one or more additional user responses to the task from the one or more additional users; and

remove the task from a set of pending tasks based on the user response, the one or more other user responses, and the one or more additional user responses collectively satisfying the consensus threshold.

7. The system of claim 1, wherein the label comprises a first semantic label that the neural network predicted for the first multimedia item.

8. The system of claim 1, wherein the label relates to a first concept that the neural network predicted to be in the first multimedia item.

9. The system of claim 1, wherein the task comprises providing user input indicating whether the prediction of the label as a corresponding label for the first multimedia item is accurate.

10. The system of claim 1, wherein the task comprises providing user input supplementing the label with additional context information.

11. A method for facilitating training of a prediction model to associate context information with multimedia items, the method being implemented by one or more processors executing computer program instructions that, when executed, perform the method, the method comprising:

providing multimedia items to a prediction model to cause the prediction model to predict one or more labels for the multimedia items, the prediction model predicting a label as a corresponding label for first and second multimedia items of the multimedia items;

assigning the first and second multimedia items to a group based on the prediction model predicting the label as a corresponding label for the first and second multimedia items,

generating, based on the predicted label, a task related to the predicted label, the task soliciting a user to indicate one or more of the multimedia items that are relevant to the predicted label;

causing the first multimedia item, the second multimedia item, and the task to be presented together on a user interface at a same time based on the assignment of the first and second multimedia items to the group;

obtaining, via the user interface, a user response to the task, the user response comprising (i) a first user indication of the first multimedia item as being relevant to the predicted label and (ii) a second user indication of the second multimedia item as being relevant or not relevant to the predicted label; and

providing, to the prediction model, the first and second user indications to cause the prediction model to be updated based on the first and second user indications.

12. The method of claim 11, further comprising:

providing, to one or more other prediction models, the first and second user indications to cause the one or more other prediction models to be updated based on the first and second user indications.

13. The method of claim 11, further comprising:

obtaining the user response to the task from a user;

obtaining one or more other user responses to the task from one or more other users; and

assigning the task to one or more additional users based on the user response and the one or more other user responses collectively not satisfying a consensus threshold.

14. The method of claim 13, further comprising:

obtaining one or more additional user responses to the task from the one or more additional users; and

removing the task from a set of pending tasks based on the user response, the one or more other user responses, and the one or more additional user responses collectively satisfying the consensus threshold.

15. The method of claim 11, wherein each of the multimedia items comprises at least one of an image, an audio, or a video, and the prediction model comprises a machine learning model.

16. A system for facilitating training of a prediction model to associate context information with multimedia items, the system comprising:

a computer system comprising one or more processors programmed with computer program instructions that, when executed, cause the computer system to;

provide multimedia items to a prediction model to cause the prediction model to predict one or more labels for the multimedia items, the prediction model predicting a label as a corresponding label for first and second multimedia items of the multimedia items;

assign the first and second multimedia items to a group based on the prediction model predicting the label as a corresponding label for the first and second multimedia items

generate, based on the predicted label, a task related to the predicted label, the task soliciting a user to indicate one or more of the multimedia items that are relevant to the predicted label;

cause the first multimedia item, the second multimedia item, and the task to be presented together on a user interface at a same time based on the assignment of the first and second multimedia items to the group;

obtain, via the user interface, a user response to the task, the user response comprising (i) a first user indication of the first multimedia item as being relevant to the predicted label and (ii) a second user indication of the second multimedia item as being relevant or not relevant to the predicted label; and

provide, to the prediction model, the first and second user indications to cause the prediction model to be updated based on the first and second user indications.

17. The system of claim 16, wherein the computer system is caused to:

provide, to one or more other prediction models, the first and second user indications to cause the one or more other prediction models to be updated based on the first and second user indications.

18. The system of claim 16, wherein the computer system is caused to:

obtain the user response to the task from a user;

obtain one or more other user responses to the task from one or more other users; and

assign the task to one or more additional users based on the user response and the one or more other user responses collectively not satisfying a consensus threshold.

19. The system of claim 18, wherein the computer system is caused to:

obtain one or more additional user responses to the task from the one or more additional users; and

remove the task from a set of pending tasks based on the user response, the one or more other user responses, and the one or more additional user responses collectively satisfying the consensus threshold.