Automatic response suggestions based on images received in messaging applications

• US 10,862,836 B2
• Filed: 09/04/2019
• Issued: 12/08/2020
• Est. Priority Date: 09/20/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: McGregor, Jr., John Patrick, Cassidy, Ryan, Fuxman, Ariel, Ramavajjala, Vivek, Ravi, Sujith, Nazarov, Sergey, Fulay, Amit
• Primary Examiner(s): Jacobs, Lashonda T
• Application Number:
  US16/560,815
• Publication Number:
  US 20190394153A1
• Time in Patent Office: 461 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/52 for supporting social networking services

Abstract

Implementations relate to automatic response suggestions based on images received in messaging applications. In some implementations, a computer-executed method includes detecting a first image included within a first message received at a second device over a communication network from a first device of a first user, and programmatically analyzing the first image to extract a first image content. The method includes retrieving a first semantic concept associated with the first image content, programmatically generating a suggested response to the first message based on the first semantic concept, and transmitting instructions causing rendering of the suggested response in the messaging application as a suggestion to a second user of the second device.

Claims

1. A computer-executed method to automatically provide suggestions in a messaging application, the computer-executed method comprising:

receiving one or more messages including a first image and a second image at a second device from a first device over a communication network;

programmatically analyzing the first image to extract a first image content;

retrieving a first semantic concept associated with the first image content;

programmatically analyzing the second image to extract a second image content;

retrieving a second semantic concept associated with the second image content;

programmatically generating a suggested response to the one or more messages, wherein the suggested response is based on the first semantic concept and the second semantic concept; and

transmitting a command to cause rendering of the suggested response in the messaging application on the second device.

2. The computer-executed method of claim 1, wherein the first semantic concept includes first text and the second semantic concept includes second text, and wherein the suggested response includes at least a portion of the first text from the first semantic concept and at least a portion of the second text from the second semantic concept.

3. The computer-executed method of claim 1, wherein the suggested response includes a suggested image, wherein the suggested image depicts a first feature related to the first semantic concept and a second feature related to the second semantic concept.

4. The computer-executed method of claim 1, further comprising:

upon receiving a selection of the suggested response based on input received at the second device, transmitting the suggested response over the communication network to the first device.

5. The computer-executed method of claim 1, further comprising:

detecting a first textual content within the one or more messages, wherein programmatically generating the suggested response is further based on the first textual content of the one or more messages.

6. The computer-executed method of claim 5, further comprising:

programmatically analyzing the first textual content to retrieve a third semantic concept,

wherein programmatically generating the suggested response to the one or more messages is further based on the third semantic concept.

7. The computer-executed method of claim 1, further comprising:

determining that a fourth semantic concept is associated with the first semantic concept; and

determining second textual content based on an association of the second textual content with the fourth semantic concept, wherein the suggested response includes the second textual content.

8. The computer-executed method of claim 1, wherein programmatically generating the suggested response further comprises:

determining a respective score for each of a plurality of suggested responses; and

selecting, based on the respective scores for the plurality of suggested responses, a particular response from the plurality of suggested responses as the suggested response.

9. The computer-executed method of claim 1, further comprising filtering previous user responses to a plurality of previous images such that filtered previous user responses are statistically associated with the plurality of previous images,

wherein programmatically generating the suggested response is based on the filtered previous user responses to the plurality of previous images.

10. The computer-executed method of claim 1, wherein retrieving the first semantic concept comprises querying a hierarchical taxonomy of concepts based on the first image content.

11. The computer-executed method of claim 1, wherein programmatically generating the suggested response comprises generating the suggested response using at least one of a graph-based learning model or one or more rules-based grammars.

12. The computer-executed method of claim 1, wherein the suggested response includes textual content, and further comprising:

determining that a particular semantic concept is associated with the first semantic concept based on a predetermined relationship between the first semantic concept and the particular semantic concept in a stored taxonomy; and

determining the textual content based on an association of the textual content with the particular semantic concept.

13. The computer-executed method of claim 1, wherein generating the suggested response includes determining the suggested response using a stored graph that indicates relationships between a plurality of concept nodes representing concepts, a plurality of image nodes representing images, and a plurality of response nodes representing responses, wherein the first semantic concept is matched to a first concept node and wherein the suggested response is determined based on strengths of connections between the first concept node and at least one of the response nodes.

14. The computer-executed method of claim 1, wherein generating the suggested response includes using a grammar-based model in which the first semantic concept is matched to one or more concepts in a stored taxonomy and one or more related concepts are determined in the stored taxonomy, and wherein the one or more related concepts are input to one or more stored grammars associated with the one or more related concepts in the grammar-based model to generate the suggested response.

15. A system to automatically suggest content in a messaging application comprising:

a memory storing instructions; and

at least one processor configured to access the instructions in the memory to perform operations comprising;

receiving one or more messages including a first image and a second image at a second device from a first device over a communication network;

posting, within the messaging application displayed on a second device, the one or more messages including the first image and the second image transmitted by the first device;

programmatically generating a suggested response to the one or more messages, the suggested response generated based on a first semantic concept associated with a first image content in the first image and a second semantic concept associated with a second image content in the second image; and

transmitting a command to cause rendering of the suggested response in the messaging application as a suggested response to the one or more messages.

16. The system of claim 15 wherein the at least one processor is configured to perform additional operations comprising:

receiving a selection of the suggested response in the messaging application based on user input provided via the second device; and

transmitting the selected suggested response to the first device.

17. The system of claim 15, wherein the first semantic concept and the second semantic concept each include text, and wherein the suggested response includes the text from the first semantic concept and the text from the second semantic concept.

18. The system of claim 15, wherein the suggested response includes a suggested image rendered in the messaging application, wherein the suggested image depicts a first feature related to the first semantic concept and a second feature related to the second semantic concept.

19. A non-transitory computer readable medium having stored thereon software instructions that, when executed by a processor, cause the processor to automatically suggest content in a messaging application by performing operations including:

receiving one or more messages including a first image and a second image at a second device from a first device over a communication network;

posting, within the messaging application displayed on a second device, the one or more messages including the first image and the second image;

programmatically generating one or more suggested responses to the one or more messages, the one or more suggested responses each generated based on a first semantic concept associated with a first image content in the first image and a second semantic concept associated with a second image content in the second image;

transmitting a command to cause rendering of at least one suggested response of the one or more suggested responses in the messaging application;

receiving a selection of a selected suggested response of the at least one suggested response in the messaging application based on user input via the second device; and

in response to receiving the selection of the selected suggested response, transmitting the selected suggested response over the communication network to the first device as a response to the one or more messages.

20. The non-transitory computer readable medium of claim 19, wherein a particular suggested response of the one or more suggested responses includes at least one of:

first text from the first semantic concept and second text from the second semantic concept;

or

a suggested image that depicts a first feature representing the first semantic concept and a second feature representing the second semantic concept.