INTELLIGENT AUTOMATED ASSISTANT IN A MESSAGING ENVIRONMENT

• US 20170132019A1
• Filed: 05/10/2016
• Published: 05/11/2017
• Est. Priority Date: 11/06/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: GRUBER, Thomas R., KARASHCHUK, Petr, VEGA GALVEZ, Tomas A.
• Granted Patent: US 10,691,473 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 2203/04105 Pressure sensors for measuring the pressure or force exerted on the touch surface without providing the touch position
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: G06Q 10/109 Time management, e.g. calendars, reminders, meetings or time accounting
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/216 Handling conversation history, e.g. grouping of messages in sessions or threads
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements

Abstract

Systems and processes for operating an intelligent automated assistant in a messaging environment are provided. In one example process, a graphical user interface (GUI) having a plurality of previous messages between a user of the electronic device and the digital assistant can be displayed on a display. The plurality of previous messages can be presented in a conversational view. User input can be received and in response to receiving the user input, the user input can be displayed as a first message in the GUI. A contextual state of the electronic device corresponding to the displayed user input can be stored. The process can cause an action to be performed in accordance with a user intent derived from the user input. A response based on the action can be displayed as a second message in the GUI.

Claims

1. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, cause the one or more processors to:

display, on a display, a graphical user interface (GUI) having a plurality of previous messages between a user of an electronic device and a digital assistant implemented on the electronic device, the plurality of previous messages presented in a conversational view;

receive user input;

in response to receiving the user input;

display the user input as a first message in the GUI; and

store a contextual state of the electronic device corresponding to the displayed user input;

cause an action to be performed in accordance with a user intent derived from the user input; and

display a response as a second message in the GUI, the response based on the action.

2. The non-transitory computer-readable medium of claim 1, wherein each of the plurality of previous messages are associated with a previous contextual state of the electronic device.

3. The non-transitory computer-readable medium of claim 1, wherein the instructions further cause the one or more processors to:

prior to displaying the user input, detect a user selection of a first previous message of the plurality of previous messages, the first previous message associated with a first previous contextual state of the electronic device, wherein causing the action to be performed further comprises;

obtaining results using the first previous contextual state of the electronic device, the results based on the user intent.

4. The non-transitory computer-readable medium of claim 1, further comprising:

prior to receiving the user input;

detecting a second user input;

in response to detecting the second user input, sampling audio data containing a user utterance;

causing a second action to be performed in accordance with a second user intent derived from the user utterance; and

outputting a second response based on the second action, wherein a second previous message of the plurality of previous messages includes a text representation of the user utterance, and wherein a third previous message of the plurality of previous messages is based on the second response.

5. The non-transitory computer-readable medium of claim 1, further comprising:

receiving a third user input associated with a fourth previous message of the plurality of previous messages;

in response to receiving the third user input, storing a bookmark of the fourth previous message in association with an affordance; and

displaying the affordance on the GUI, wherein selecting the affordance causes a focus of the GUI to change to the fourth previous message.

6. The non-transitory computer-readable medium of claim 1, further comprising:

in response to displaying the user input as a first message in the GUI of the electronic device, causing display of the first message in a GUI of a second electronic device; and

in response to displaying the response as a second message in the GUI of the electronic device, causing display of the second message in the GUI of the second electronic device.

7. The non-transitory computer-readable medium of claim 6, further comprising:

at the second electronic device;

detecting a user selection of the first message in the GUI of the second electronic device, the first message associated with the contextual state of the electronic device;

receiving a fourth user input;

in response to receiving the fourth user input, displaying the fourth user input as a third message in the GUI of the second electronic device;

causing a third action to be performed in accordance with a third user intent, the third user intent derived from the fourth user input and the contextual state of the electronic device; and

displaying a third response as a fourth message in the GUI of the second electronic device, the third response based on the third action.

8. The non-transitory computer-readable medium of claim 1, further comprising:

identifying, based on the contextual state of the electronic device and the derived user intent, one or more words in the displayed response corresponding to an entity;

determining an action associated with the entity; and

enabling selection of the one or more words in the displayed response, wherein detecting a user selection of the one or more words causes the action associated with the entity to be performed.

9. The non-transitory computer-readable medium of claim 1, wherein the GUI of the electronic device includes a text input field, and further comprising:

detecting a fifth user input associated with a first previous message of the plurality of previous messages; and

in response to detecting the fifth user input, displaying text of the first previous message in the text input field of the GUI.

10. The non-transitory computer-readable medium of claim 1, further comprising:

receiving, via the GUI of the electronic device, a user selection of the second message;

in response to receiving the user selection of the second message;

obtaining, using an application of the electronic device, detailed results based on the user intent; and

displaying the detailed result in a GUI of the application.

11. The non-transitory computer-readable medium of claim 10, wherein displaying the response as the second message includes displaying an indication in the second message that a more detailed response is available in response to selecting the second message.

12. The non-transitory computer-readable medium of claim 1, further comprising:

displaying, based on the user intent, a media object in the GUI of the electronic device, the media object associated with an application of the electronic device.

13. The non-transitory computer-readable medium of claim 12, further comprising:

detecting a user selection of the media object;

in response to detecting the user selection of the media object;

obtaining, using an application of the electronic device, expanded results corresponding to the media object; and

displaying the expanded results in a GUI of the application.

14. The non-transitory computer-readable medium of claim 1, further comprising:

receiving a notification from an application of the electronic device; and

in response to receiving the notification, displaying the notification as a fifth message in the GUI.

15. The non-transitory computer-readable medium of claim 1, wherein the GUI of the electronic device includes a user input interface, wherein the displayed response includes a request for additional information, and further comprising:

displaying two or more suggested character strings in the user input interface, each of the two or more suggested character string responsive to the request for additional information;

detecting a user selection of a suggested character string of the two or more suggested character strings;

in response to detecting a user selection of a suggested character string, displaying the selected suggested character string as a sixth message in the GUI of the electronic device; and

displaying a fourth response as a seventh message in the GUI of the electronic device, the fourth response based on the selected suggested character strings.

16. The non-transitory computer-readable medium of claim 1, wherein:

the user input includes an ambiguous term;

the displayed response includes a displayed request for additional information related to the ambiguous term; and

the displayed request for additional information includes a list of possible interpretations for the ambiguous term.

17. The non-transitory computer-readable medium of claim 16, further comprising:

receiving a sixth user input indicating a selection of one of the list of possible interpretations;

in response to receiving the sixth user input, displaying the sixth user input as an eighth message in the GUI;

causing a sixth action to be performed in accordance with a user intent derived from the sixth user input; and

displaying a sixth response as a ninth message in the GUI, the sixth response based on the sixth action.

18. The non-transitory computer-readable medium of claim 1, wherein the contextual state includes a time the user input is received and data defining the location of the electronic device at the time the user input is received.

19. A method for operating a digital assistant, the method comprising:

at an electronic device with a display, a processor, and memory;

displaying, on the display, a graphical user interface (GUI) having a plurality of previous messages between a user of the electronic device and the digital assistant, the plurality of previous messages presented in a conversational view;

receiving user input;

in response to receiving the user input;

displaying the user input as a first message in the GUI; and

storing a contextual state of the electronic device corresponding to the displayed user input;

causing an action to be performed in accordance with a user intent derived from the user input; and

displaying a response as a second message in the GUI, the response based on the action.

20. An electronic device comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the one or more processors to;

display, on a display, a graphical user interface (GUI) having a plurality of previous messages between a user of an electronic device and a digital assistant implemented on the electronic device, the plurality of previous messages presented in a conversational view;

receive user input;

in response to receiving the user input;

display the user input as a first message in the GUI; and

store a contextual state of the electronic device corresponding to the displayed user input;

cause an action to be performed in accordance with a user intent derived from the user input; and

display a response as a second message in the GUI, the response based on the action.