Personality reply for digital content

• US 10,341,272 B2
• Filed: 05/05/2017
• Issued: 07/02/2019
• Est. Priority Date: 05/05/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Badr, Ibrahim, Kumar, Aayush, Bakir, Goekhan Hasan, Grimsmo, Nils, Buisman, Bianca Madalina
• Primary Examiner(s): Fan, Hua
• Application Number:
  US15/587,783
• Publication Number:
  US 20180324117A1
• Time in Patent Office: 788 Days
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/10 Multimedia information

Abstract

A computer-implemented method is described. The method includes a computing system receiving an item of digital content from a user device. The computing system generates one or more labels that indicate attributes of the item of digital content. The computing system also generates one or more conversational replies to the item of digital content based on the one or more labels that indicate attributes of the item of digital content. The method also includes the computing system selecting a conversational reply from among the one or more conversational replies and providing the conversational reply for output to the user device.

Claims

1. A computer-implemented method, comprising:

receiving, by an automated conversational assistant system that is configured to conduct an automated conversation with a user and from a user device, an image;

generating, by the automated conversational assistant system, one or more image labels and a corresponding score for each image label, wherein each image label is descriptive of a feature depicted in the image and each score indicates a prominence of the feature relative to other features depicted in the image;

selecting, by the automated conversational assistant system and from the one or more image labels, a particular image label that corresponds to a prominent feature depicted in the image based on the score for the particular image label exceeding a threshold score;

based on the particular image label that corresponds to the prominent feature of the image, generating, by the automated conversational assistant system, one or more conversational replies to the image using past conversations between multiple users, predetermined conversational replies, and quotations from media content;

selecting, by the automated conversational assistant system and from among the one or more conversational replies, a conversational reply based on a relationship between content of each of the one or more conversational replies and the particular image label; and

providing, for output by the automated conversational assistant system and to the user device, the conversational reply.

2. The method of claim 1, wherein generating the one or more conversational replies comprises:

determining, by at least one module, a similarity score that indicates a similarity between the image and an item of digital content that is associated with an electronic conversation; and

responsive to the similarity score exceeding a threshold similarity score, generating, by the at least one module and based on the item of digital content, the one or more conversational replies and a confidence score for each conversational reply of the one or more conversational replies; and

wherein selecting the conversational reply comprises;

selecting, from among the one or more conversational replies, a particular conversational reply based on the confidence score for the particular conversational reply.

3. The method of claim 1, wherein generating the one or more conversational replies comprises:

analyzing, by at least one module, content of a content database comprising multiple items of media content, wherein the content is associated with at least one item of media content;

determining, by the at least one module, a similarity score that indicates a similarity between the particular image label and the analyzed content; and

responsive to the similarity score exceeding a threshold similarity score, generating, by the at least one module and based on a particular item of media content, the one or more conversational replies and a confidence score for each conversational reply of the one or more conversational replies; and

wherein selecting the conversational reply comprises;

selecting, from among the one or more conversational replies, a particular conversational reply based on the confidence score for the particular conversational reply.

4. The method of claim 1, wherein generating the one or more conversational replies comprises:

determining, by at least one module, a first similarity score that indicates a similarity between the particular image label and a predetermined conversational reply of a reply content database; and

responsive to the first similarity score exceeding a threshold similarity score, generating, by the at least one module and based on a particular predetermined conversational reply, the one or more conversational replies and a confidence score for each conversational reply of the one or more conversational replies; and

wherein selecting the conversational reply comprises;

selecting, from among the one or more conversational replies, a particular conversational reply based on the confidence score for the particular conversational reply.

5. The method of claim 4, wherein generating the one or more conversational replies further comprises:

determining, by the at least one module and based on the particular image label, a related entity that has a threshold relevance to the image, wherein the related entity is determined using a knowledge graph;

determining, by the at least one module, a second similarity score that indicates a similarity between the particular image label, the determined related entity, and the particular predetermined conversational reply; and

responsive to the second similarity score exceeding a threshold similarity score, selecting by the at least one module, a predetermined conversational reply for inclusion with the one or more conversational replies generated by the automated conversational assistant system.

6. The method of claim 5, wherein obtaining the related entity that has the threshold relevance to the image comprises:

accessing, by the automated conversational assistant system, the knowledge graph that includes nodes that each represent an entity and edges that each represent a relationship between a pair of nodes; and

obtaining, by the automated conversational assistant system and from the knowledge graph, the related entity that has the threshold relevance to the image based on a similarity between the particular image label and entity-pairs that correspond to nodes and edges of the knowledge graph.

7. The method of claim 6, wherein obtaining the related entity that has the threshold relevance to the image comprises:

determining, by the knowledge graph, a measure of similarity between the particular image label and multiple distinct entity-pairs that each correspond to a node and edge of the knowledge graph, wherein the measure of similarity indicates a relevance of each entity-pair to the image; and

identifying, by the knowledge graph, the related entity of an entity-pair having a corresponding measure of similarity that indicates the related entity has the threshold relevance to the image.

8. The method of claim 1, wherein generating the one or more conversational replies comprises at least one of:

generating, by a first module of the automated conversational assistant system, a first conversational reply and generating a first confidence score for the first conversational reply; and

generating, by a second module of the automated conversational assistant system, a second conversational reply and generating a second confidence score for the second conversational reply.

9. The method of claim 8, wherein generating the first confidence score for the first conversational reply by the first module comprises:

determining a first relevance parameter that indicates a relevance between the first conversational reply and the image; and

generating the first confidence score based on the first relevance parameter.

10. The method of claim 8, wherein generating the second confidence score for the second conversational reply by the second module comprises:

determining a second relevance parameter that indicates a relevance between the second conversational reply and the particular image label; and

generating the second confidence score based on the second relevance parameter.

11. The method of claim 1, wherein the method further comprises:

selecting a particular number of conversational replies from among the one or more conversational replies based on respective confidence scores of the one or more conversational replies; and

providing, for output by the user device, the particular number of conversational replies as a suggested reply to the image received by the automated conversational assistant system.

12. The method of claim 11, wherein providing the conversational reply selected from among the one or more conversational replies comprises:

the automated conversational assistant system receiving the image as a communication message of an electronic conversation generated by a messaging application, the electronic conversation including at least one user;

providing the particular number of conversational replies as a suggested reply message to the communication message of the electronic conversation; and

receiving, by the automated conversational assistant system, an indication that the at least one user selected a conversational reply from among the particular number of conversational replies as a reply message to the communication message of the electronic conversation.

13. The method of claim 1, wherein generating the one or more image labels and the corresponding score for each image label comprises:

generating a respective boundary box for one or more features depicted in the image, each boundary box being configured to bound a respective feature in the image; and

generating an image label and a corresponding score for each feature that is bounded by a respective boundary box, wherein the corresponding score indicates a prominence of the feature based on a relevance of the image label to the feature bounded by a respective boundary box.

14. An electronic system comprising:

one or more processing devices;

one or more non-transitory machine-readable storage devices for storing instructions that are executable by the one or more processing devices to cause performance of operations comprising;

receiving, by an automated conversational assistant system that is configured to conduct an automated conversation with a user and from a user device, an image;

generating, by the automated conversational assistant system, one or more image labels and a corresponding score for each image label, wherein each image label is descriptive of a feature depicted in the image and each score indicates a prominence of the feature relative to other features depicted in the image;

selecting, by the automated conversational assistant system and from the one or more image labels, a particular image label that corresponds to a prominent feature depicted in the image based on the score for the particular image label exceeding a threshold score;

based on the particular image label that corresponds to the prominent feature of the image, generating, by the automated conversational assistant system, one or more conversational replies to the image using past conversations between multiple users, predetermined conversational replies, and quotations from media content;

selecting, by the automated conversational assistant system and from among the one or more conversational replies, a conversational reply based on a relationship between content of each of the one or more conversational replies and the particular image label; and

providing, for output by the automated conversational assistant system and to the user device, the conversational reply.

15. The electronic system of claim 14, wherein generating the one or more conversational replies comprises:

determining, by at least one module, a similarity score that indicates a similarity between the image and an item of digital content that is associated with an electronic conversation; and

responsive to the similarity score exceeding a threshold similarity score, generating, by the at least one module and based on the item of digital content, the one or more conversational replies and a confidence score for each conversational reply of the one or more conversational replies; and

wherein selecting the conversational reply comprises;

selecting, from among the one or more conversational replies, a particular conversational reply based on the confidence score for the particular conversational reply.

16. The electronic system of claim 14, wherein generating the one or more conversational replies comprises:

analyzing, by at least one module, content of a content database comprising multiple items of media content, wherein the content is associated with at least one item of media content; and

determining, by the at least one module, a similarity score that indicates a similarity between the particular image label and the analyzed content; and

responsive to the similarity score exceeding a threshold similarity score, generating, by the at least one module and based on a particular item of media content, the one or more conversational replies and a confidence score for each conversational reply of the one or more conversational replies; and

wherein selecting the conversational reply comprises;

selecting, from among the one or more conversational replies, a particular conversational reply based on the confidence score for the particular conversational reply.

17. The electronic system of claim 14, wherein generating the one or more conversational replies comprises:

determining, by at least one module, a first similarity score that indicates a similarity between the particular image label and a predetermined conversational reply of a reply content database; and

responsive to the first similarity score exceeding a threshold similarity score, generating, by the at least one module and based on a particular predetermined conversational reply, the one or more conversational replies and a confidence score for each conversational reply of the one or more conversational replies; and

wherein selecting the conversational reply comprises;

selecting, from among the one or more conversational replies, a particular conversational reply based on the confidence score for the particular conversational reply.

18. The electronic system of claim 17, wherein generating the one or more conversational replies further comprises:

determining, by the at least one module and based on the particular image label, a related entity that has a threshold relevance to the image, wherein the related entity is determined using a knowledge graph;

determining, by the at least one module, a second similarity score that indicates a similarity between the particular image label, the determined related entity, and the particular predetermined conversational reply; and

responsive to the second similarity score exceeding a threshold similarity score, selecting by the at least one module, a predetermined conversational reply for inclusion with the one or more conversational replies generated by the automated conversational assistant system.

19. The electronic system of claim 14, wherein generating the one or more conversational replies comprises at least one of:

generating, by a first module of the automated conversational assistant system, a first conversational reply and generating a first confidence score for the first conversational reply; and

generating, by a second module of the automated conversational assistant system, a second conversational reply and generating a second confidence score for the second conversational reply.

20. One or more non-transitory machine-readable storage devices storing instructions that are executable by one or more processing devices to cause performance of operations comprising:

receiving, by an automated conversational assistant system that is configured to conduct an automated conversation with a user and from a user device, an image;

generating, by the automated conversational assistant system, one or more image labels and a corresponding score for each image label, wherein each image label is descriptive of a feature depicted in the image and each score indicates a prominence of the feature relative to other features depicted in the image;

selecting, by the automated conversational assistant system and from the one or more image labels, a particular image label that corresponds to a prominent feature depicted in the image based on the score for the particular image label exceeding a threshold score;

based on the particular image label that corresponds to the prominent feature of the image, generating, by the automated conversational assistant system, one or more conversational replies to the image using past conversations between multiple users, predetermined conversational replies, and quotations from media content;

selecting, by the automated conversational assistant system and from among the one or more conversational replies, a conversational reply based on a relationship between content of each of the one or more conversational replies and the particular image label; and

providing, for output by the automated conversational assistant system and to the user device, the conversational reply.