Conversation user interface

• US 9,223,537 B2
• Filed: 04/18/2012
• Issued: 12/29/2015
• Est. Priority Date: 04/18/2012
• Status: Active Grant

• Current Assignee: Verint Americas Incorporated (Verint Systems Incorporated)
• Original Assignee: Next IT Corporation (Verint Systems Incorporated)
• Inventors: Brown, Fred A, Miller, Tanya M, Culley, Bryan Michael, Snavely, Eli D., Wooters, Charles C
• Primary Examiner(s): Gutierrez, Andres E
• Application Number:
  US13/449,927
• Publication Number:
  US 20130283168A1
• Time in Patent Office: 1,350 Days
• Field of Search: 715/728
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04L 51/04 Real-time or near real-time messaging, e.g. instant messaging [IM]

Abstract

A conversation user interface enables users to better understand their interactions with computing devices, particularly when speech input is involved. The conversation user interface conveys a visual representation of a conversation between the computing device, or virtual assistant thereon, and a user. The conversation user interface presents a series of dialog representations that show input from a user (verbal or otherwise) and responses from the device or virtual assistant. Associated with one or more of the dialog representations are one or more graphical elements to convey assumptions made to interpret the user input and derive an associated response. The conversation user interface enables the user to see the assumptions upon which the response was based, and to optionally change the assumption(s). Upon change of an assumption, the conversation GUI is refreshed to present a modified dialog representation of a new response derived from the altered set of assumptions.

Claims

1. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed on one or more processors, cause the one or more processors of a virtual assistant service to perform acts comprising:

generating data for a conversation graphical user interface (GUI) that represents a virtual assistant;

causing display of the conversation GUI via a computing device to enable a conversation between the virtual assistant and a user of the computing device;

receiving user input that is provided via the conversation GUI, the user input comprising one of audio input, keypad input, or touch input;

parsing the user input with a natural language processing system that employs a language model;

determining, with the natural language processing system, a response based at least in part on (i) the parsed user input and (ii) at least one of content of a service provider, content of the virtual assistant service, or content of the computing device of the user;

identifying an assumption that is used to determine the response, the assumption comprising at least one of the language model that is employed by the natural language processing system, a profile for the user, or a learned behavior of the user;

causing display of a dialog representation in the conversation GUI for the user input;

causing display of a dialog representation in the conversation GUI for the response;

causing display of the assumption in the conversation GUI;

enabling the user to interact with the conversation GUI to modify the assumption;

receiving a modification to the assumption;

determining a revised response based at least in part on the modification to the assumption; and

causing display of the revised response in the conversation GUI.

2. One or more non-transitory computer-readable media as recited in claim 1, wherein the user input is the audio input, and the user input is represented in the conversation GUI in a text format.

3. One or more non-transitory computer-readable media as recited in claim 1, wherein the user input is the audio input, and the determining a response comprises a combination of interpreting the audio input and ascertaining a suitable response for the audio input as interpreted.

4. One or more non-transitory computer-readable media as recited in claim 3, wherein the assumption further includes an assumption pertaining to interpretation of the audio input.

5. One or more non-transitory computer-readable media as recited in claim 1, wherein the acts further comprise learning, from past user behavior, what assumptions should be used in making the determination of the response.

6. One or more non-transitory computer-readable media as recited in claim 1, wherein the acts further comprise identifying a context associated with the user or associated with a session of the user on a site, wherein the assumption used to determine the response is set, at least in part, by the context.

7. One or more non-transitory computer-readable media as recited in claim 1, wherein the acts further comprise recording modifications made by the user to the assumption and learning from the user modifications to adjust how the determination of the response to a future user input is made.

8. One or more non-transitory computer-readable media as recited in claim 1, wherein the acts further comprise enabling the user to interact with the conversation GUI to select a prior dialog bubble and start a new conversation from the prior dialog bubble.

9. One or more non-transitory computer-readable media as recited in claim 1, wherein the acts further comprise

identifying one or more concepts expressed in the user input;

identifying a context associated with the user or associated with a session of the user on a site; and

determining an intent of the user input based at least in part on the one or more concepts and the context; and

wherein the response is determined based at least in part on the intent and the context.

10. One or more non-transitory computer-readable media as recited in claim 9, wherein the user input comprises a string of text, and the identifying of the one or more concepts comprises identifying one or more keywords within the string of text and mapping the one or more keywords to the one or more concepts.

11. One or more non-transitory computer-readable media as recited in claim 9, wherein the user input comprises a string of text, and the identifying of the one or more concepts comprises:

identifying one or more tokens within the string of text;

mapping the one or more tokens to one or more vocab items; and

pattern matching the one or more vocab items to the one or more concepts.

12. One or more non-transitory computer-readable media as recited in claim 9, wherein the determining of the intent comprises mapping the one or more concepts to one of multiple different intents associated with the one or more concepts based at least in part on the context.

13. One or more non-transitory computer-readable media as recited in claim 9, wherein the determining of the response comprises mapping the intent to one of multiple different responses associated with the intent based at least in part on the context.

14. One or more non-transitory computer-readable media as recited in claim 9, wherein the intent is identified with reference to at least a first portion of the context and the response is identified with reference to at least a second, different portion of the context.

15. One or more non-transitory computer-readable media as recited in claim 9, wherein the context comprises:

whether or not the user has signed in with the site of the service provider;

a status of the user with the service provider;

a page of the site from which the user provides the user input to the virtual assistant;

how long the user has remained on the page of the site from which the user provides the user input to the virtual assistant;

a navigation history of the user during the session prior to the user providing the user input to the virtual assistant;

a location of a cursor when the user provides the user input to the virtual assistant;

a prior query provided by the user to the virtual assistant during the session or during a prior session with the site;

a time of day at which the user provides the user input to the virtual assistant;

a date on which the user provides the user input to the virtual assistant;

an age of the user;

a location of the user;

a device type from which the user accesses the site;

a language of the user;

user preferences;

user buying patterns;

past user behavior;

or

a previous interaction between the user and the virtual assistant.

16. One or more non-transitory computer-readable media as recited in claim 9, wherein the acts further comprise causing an action on behalf of the user, the action comprising one of:

purchasing an item on behalf of the user;

initiating a request on behalf of the user;

initiating a communication on behalf of the user;

communication to others on behalf of the user;

or

altering a reservation or order on behalf of the user.

17. One or more non-transitory computer-readable media as recited in claim 9, wherein the acts further comprise learning over time that certain words of the user input map to certain vocabulary terms.

18. One or more non-transitory computer-readable media as recited in claim 9, wherein the acts further comprise learning over time that certain phrases of the user input map to corresponding concepts.

19. A portable computing device comprising:

one or more non-transitory computer-readable media as recited in claim 1; and

one or more processors to execute the computer-executable instructions stored on the computer-readable media to perform the acts.

20. A computing system comprising:

one or more servers for hosting a site;

the one or more non-transitory computer-readable media as recited in claim 1; and

one or more processors to execute the computer-executable instructions stored on the computer-readable media to perform the acts.

21. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed on one or more processors, cause the one or more processors to perform acts comprising:

parsing user input with natural language processing techniques that utilize a language model, the user input comprising at least one of gesture input, audio input, keypad input, or touch input;

determining a response based at least in part on the parsed user input;

causing display of a conversation user interface having a first dialog representation associated with the user input and a second dialog representation associated with the response, the conversation user interface being associated with a virtual assistant;

providing, for display as part of the conversation user interface, one or more graphical elements to convey at least one assumption made to interpret the user input and determine the response; and

enabling, via the conversation user interface, a user to change the assumption, whereupon change of the assumption causes the second dialog representation associated with the response to be removed and a third dialog representation to be displayed, the third dialog representation representing an updated response for the virtual assistant.

22. One or more non-transitory computer-readable media as recited in claim 21, wherein the assumption comprises a speech recognition assumption that pertains to recognizing verbal input from the user.

23. One or more non-transitory computer-readable media as recited in claim 21, wherein the assumption comprises a logic assumption that pertains to determining the response to the user input.

24. One or more non-transitory computer-readable media as recited in claim 21, wherein the assumption comprises one or more assistant-initiated clarifications that ask the user to clarify the user input.

25. One or more non-transitory computer-readable media as recited in claim 21, wherein one or more assumptions are associated with each of the first and second dialog representations.

26. One or more non-transitory computer-readable media as recited in claim 21, wherein the one or more graphical elements that convey associated assumptions are initially hidden in the conversation user interface, the first and second dialog representations being independently selectable via the conversation user interface to reveal the one or more graphical elements in association with a corresponding first or second dialog representation.

27. One or more non-transitory computer-readable media as recited in claim 21, wherein the user has a user profile, and the acts further comprise determining the assumption made to interpret the user input based at least in part on the user profile.

28. One or more non-transitory computer-readable media as recited in claim 21, wherein the acts further comprise determining the assumption made to interpret the user input based at least in part on the user profile.

29. One or more non-transitory computer-readable media as recited in claim 21, wherein the acts further comprise determining the assumption made to interpret the user input based at least in part a context associated with the user.

30. One or more non-transitory computer-readable media as recited in claim 21, wherein the acts further comprise learning characteristics of the user, and determining the assumption made to interpret the user input based at least in part these learned characteristics.

31. One or more non-transitory computer-readable media as recited in claim 30, wherein the characteristics of the user comprises one or more of speech features, preferences, behaviors, patterns, and navigation history.

32. One or more non-transitory computer-readable media as recited in claim 21, wherein the acts further comprise enabling removal of one or more of the first or second dialog representations from the conversation user interface.

33. One or more non-transitory computer-readable media as recited in claim 21, wherein the acts further comprise audibly outputting the response.

34. One or more non-transitory computer-readable media as recited in claim 21, wherein the acts further comprise:

identifying one or more concepts expressed in the user input;

identifying a context associated with the user or associated with a session of the user on a site; and

determining an intent of the user input based at least in part on the one or more concepts and the context; and

wherein the response to the user input is determined based at least in part on the intent and the context.

35. A portable computing device comprising:

one or more non-transitory computer-readable media as recited in claim 21; and

one or more processors to execute the computer-executable instructions stored on the computer-readable media to perform the acts.

36. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed on one or more processors, cause the one or more processors to perform acts comprising:

causing display of a conversation user interface that represents a conversation between a user and a virtual assistant;

parsing, based at least in part on natural language processing techniques that employ a language model, input from the user that is received during the conversation between the user and the virtual assistant, the input from the user comprising at least one of gesture input, audio input, keypad input, or touch input;

determining a response for the virtual assistant based at least in part on the parsed input;

adding, for display in the conversation user interface, a first dialog representation associated with the input from the user and a second dialog representation associated with the response from the virtual assistant to visually convey the conversation; and

enabling, via interaction with the conversation user interface, the user to select a particular one of the first or second dialog representations within the conversation user interface and restart the conversation from the particular one of the first or second dialog representations, the restarting including removing the particular one of the first or second dialog representations and displaying a third dialog representation associated with an updated response from the virtual assistant.

37. A portable computing device comprising:

one or more non-transitory computer-readable media as recited in claim 36; and

one or more processors to execute the computer-executable instructions stored on the one or more non-transitory computer-readable media to perform the acts.

38. A computing device, comprising:

a memory;

at least one processor;

a display;

a communication module stored in the memory and executed by the at least one processor to;

send input from a user of the computing device to be parsed by a natural language processing system that employs a language model, the input including at least one of audio or text; and

receive from the natural language processing system a response to the input from the user; and

a conversation application stored in the memory and executed by the at least one processor to facilitate a conversation with the user of the computing device, the conversation application providing a conversation graphical user interface (GUI) to be displayed on the display, the conversation GUI comprising;

an entry portion to receive, from the user, the input;

an area to visually convey the conversation, the area to present a user dialog representation associated with the input from the user and a device dialog representation associated with the response to the input from the user; and

a control associated with at least one of the user or a device dialog representation to visually change, upon actuation, the conversation GUI to reveal an assumption that was used to determine the response, the assumption comprising at least one of a value obtained from the conversation, a value that is external to the conversation, or a value that is obtained from a previous conversation with the user.

39. The computing device of claim 38, embodied as a portable communication device.

40. A method comprising:

under control of one or more computing devices configured to implement a virtual assistant,

parsing user input with a natural language processing system that employs a language model, the user input comprising at least one of gesture input, audio input, keypad input, or touch input;

representing the user input in a conversation user interface;

determining, with the natural language processing system, a response to the user input;

representing the response to the user input in the conversation user interface;

enabling a user to interact with the conversation user interface to ascertain how the response was determined and to modify assumptions used to determine the response, wherein modification of one or more assumptions results in a modified response, and wherein individual assumptions comprise at least one of a value obtained via the conversation user interface, a value that is external to the conversation user interface, or a value that is obtained from a previous conversation between the user and the virtual assistant; and

representing the modified response in the conversation user interface.

41. A method of claim 40, wherein the representing the user input and representing the response comprises graphically presenting a series of dialog bubbles that visually distinguish between the user input and the response.

42. A method of claim 40, wherein the enabling comprises providing a control to reveal, upon actuation, how the response to the user input was determined.

43. A method of claim 40, further comprising learning characteristics of the user and choosing the assumptions used to determine the response from the characteristics.

44. A method of claim 43, wherein the characteristics of the user comprises one or more of speech features, preferences, behaviors, patterns, and navigation history.

45. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed on one or more processors, cause the one or more processors to perform acts comprising:

parsing input from a user with natural language processing techniques that utilize a language model, the input from the user comprising at least one of gesture input, audio input, keypad input, or touch input;

determining a response to the input from the user based at least in part on the parsed input;

serving or rendering a conversation graphical user interface (GUI), the conversation GUI comprising;

an entry portion to receive, from the user, the input;

an area to visually convey a conversation, the area to present a user dialog representation associated with the input received from the user and a device dialog representation associated with the response to the input from the user; and

a control associated with at least one of the user or device dialog representation to visually change, upon actuation, the conversation GUI to reveal an assumption that was used to determine the response, the assumption comprising at least one of a value obtained from the conversation, a value that is external to the conversation, or a value that is obtained from a previous conversation with the user.

46. One or more non-transitory computer-readable media as recited in claim 45, wherein the response was determined based on learned characteristics of the user.

47. One or more non-transitory computer-readable media as recited in claim 46, wherein the characteristics of the user comprises one or more of speech features, preferences, behaviors, patterns, and navigation history.