Using Context Information To Facilitate Processing Of Commands In A Virtual Assistant

• US 20120265528A1
• Filed: 09/30/2011
• Published: 10/18/2012
• Est. Priority Date: 06/05/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Gruber, Thomas Robert, Brigham, Christopher Dean, Novick, Gregory, Phipps, Benjamin S., Keen, Daniel S.
• Granted Patent: US 9,858,925 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models

Abstract

A virtual assistant uses context information to supplement natural language or gestural input from a user. Context helps to clarify the user'"'"'s intent and to reduce the number of candidate interpretations of the user'"'"'s input, and reduces the need for the user to provide excessive clarification input. Context can include any available information that is usable by the assistant to supplement explicit user input to constrain an information-processing problem and/or to personalize results. Context can be used to constrain solutions during various phases of processing, including, for example, speech recognition, natural language processing, task flow processing, and dialog generation.

Claims

1. A computer-implemented method for interpreting user input to perform a task on a computing device having at least one processor, comprising:

at a processor, receiving context information from a context source;

at an output device, prompting a user for input;

at an input device, receiving user input;

at the processor, interpreting the received user input to derive a representation of user intent;

at the processor, identifying at least one task and at least one parameter for the task, based at least in part on the derived representation of user intent;

at the processor, executing the at least one task using the at least one parameter, to derive a result;

at the processor, generating a dialog response based on the derived result; and

at the output device, outputting the generated dialog response;

wherein at least one of the steps of prompting the user for input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response is performed using the received context information.

2. The method of claim 1, wherein the received context information comprises at least one selected from the group consisting of:

application context;

personal data associated with the user;

data from a database associated with the user;

data obtained from dialog history;

data received from at least one sensor;

application preferences;

application usage history;

data describing an event;

data describing an acoustic environment in which the spoken input is received;

current dialog state;

input previously provided by the user;

location;

local time; and

environmental conditions.

3. The method of claim 1, wherein:

prompting the user comprises prompting the user via a conversational interface; and

receiving user input comprises receiving user input via the conversational interface by;

receiving spoken input; and

converting the spoken input to a text representation.

4. The method of claim 3, wherein converting the spoken input to a text representation comprises:

generating a plurality of candidate text interpretations of the spoken input; and

ranking at least a subset of the generated candidate text interpretations;

wherein at least one of the generating and ranking steps is performed using the received context information.

5. The method of claim 4, wherein the received context information used in at least one of the generating and ranking steps comprises at least one selected from the group consisting of:

data describing an acoustic environment in which the spoken input is received;

data received from at least one sensor;

vocabulary obtained from a database associated with the user;

vocabulary associated with application preferences;

vocabulary obtained from usage history; and

current dialog state.

6. The method of claim 1, wherein receiving user input comprises receiving text input.

7. The method of claim 1, wherein prompting the user comprises generating at least one prompt based at least in part on the received context information.

8. The method of claim 1, wherein interpreting the received user input to derive a representation of user intent comprises performing natural language processing on the received user input based at least in part on the received context information.

9. The method of claim 8, wherein the received context information used in interpreting the received user input comprises at least one selected from the group consisting of:

data describing an event;

application context;

input previously provided by the user;

known information about the user;

location;

date;

environmental conditions; and

history.

10. The method of claim 1, wherein performing natural language processing comprises selecting among a plurality of candidate interpretations of the received user input using the received context information.

11. The method of claim 1, wherein performing natural language processing comprises determining a referent for at least one pronoun in the received user input.

12. The method of claim 1, wherein identifying at least one task and at least one parameter for the task comprises identifying at least one task and at least one parameter for the task based at least in part on the received context information.

13. The method of claim 12, wherein identifying at least one task and at least one parameter for the task based at least in part on the received context information comprises:

receiving a plurality of candidate representations of user intent;

determining a preferred interpretation of user intent based on at least one selected from the group consisting of;

at least one domain model;

at least one task flow model; and

at least one dialog flow model.

14. The method of claim 12, wherein the received context information used in identifying at least one task and at least one parameter for the task comprises at least one selected from the group consisting of:

data describing an event;

data from a database associated with the user;

data received from at least one sensor;

application context;

input previously provided by the user;

known information about the user;

location;

date;

environmental conditions; and

history.

15. The method of claim 1, wherein generating a dialog response comprises generating a dialog response based at least in part on the received context information.

16. The method of claim 15, wherein generating a dialog response based at least in part on the received context information comprises at least one selected from the group consisting of:

generating a dialog response including a named referent;

generating a dialog response including a symbolic name associated with a telephone number;

determining which of a plurality of names to use for a referent;

determining a level of detail for the generated response; and

filtering a response based on previous output.

17. The method of claim 15, wherein the received context information used in generating a dialog response comprises at least one selected from the group consisting of:

data from a database associated with the user;

application context;

input previously provided by the user;

known information about the user;

location;

date;

environmental conditions; and

history.

18. The method of claim 1, wherein the received context information comprises at least one selected from the group consisting of:

context information stored at a server; and

context information stored at a client.

19. The method of claim 1, wherein receiving context information from a context source comprises:

requesting the context information from a context source; and

receiving the context information in response to the request.

20. The method of claim 1, wherein receiving context information from a context source comprises:

receiving at least a portion of the context information prior to receiving user input.

21. The method of claim 1, wherein receiving context information from a context source comprises:

receiving at least a portion of the context information after receiving user input.

22. The method of claim 1, wherein receiving context information from a context source comprises:

receiving static context information as part of an initialization step; and

receiving additional context information after receiving user input.

23. The method of claim 1, wherein receiving context information from a context source comprises:

receiving push notification of a change in context information; and

responsive to the push notification, updating locally stored context information.

24. The method of claim 1, wherein the computing device comprises at least one selected from the group consisting of:

a telephone;

a smartphone;

a tablet computer;

a laptop computer;

a personal digital assistant;

a desktop computer;

a kiosk;

a consumer electronic device;

a consumer entertainment device;

a music player;

a camera;

a television;

an electronic gaming unit; and

a set-top box.

25. A computer program product for interpreting user input to perform a task on a computing device having at least one processor, comprising:

a nontransitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

receiving context information from a context source;

causing an output device to prompt a user for input;

receiving user input via an input device;

interpreting the received user input to derive a representation of user intent;

identifying at least one task and at least one parameter for the task, based at least in part on the derived representation of user intent;

executing the at least one task using the at least one parameter, to derive a result;

generating a dialog response based on the derived result; and

causing the output device to output the generated dialog response;

wherein the computer program code is configured to cause at least one processor to perform at least one of the steps of causing the output device to prompt the user for input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response using the received context information.

26. The computer program product of claim 25, wherein the received context information comprises at least one selected from the group consisting of:

application context;

personal data associated with the user;

data from a database associated with the user;

data obtained from dialog history;

data received from at least one sensor;

application preferences;

application usage history;

data describing an event;

data describing an acoustic environment in which the spoken input is received;

current dialog state;

input previously provided by the user;

location;

local time; and

environmental conditions.

27. The computer program product of claim 25, wherein:

the computer program code configured to cause an output device to prompt the user comprises computer program code configured to cause an output device to prompt the user via a conversational interface; and

the computer program code configured to cause at least one processor to receive user input comprises computer program code configured to cause at least one processor to receive user input via the conversational interface.

28. The computer program product of claim 25, wherein the computer program code configured to cause at least one processor to receive user input comprises:

computer program code configured to cause at least one processor to receive spoken input; and

computer program code configured to cause at least one processor to convert the spoken input to a text representation by;

generating a plurality of candidate text interpretations of the spoken input; and

ranking at least a subset of the generated candidate text interpretations;

wherein at least one of the generating and ranking steps is performed using the received context information.

29. The computer program product of claim 28, wherein the received context information used in at least one of the generating and ranking steps comprises at least one selected from the group consisting of:

data describing an acoustic environment in which the spoken input is received;

data received from at least one sensor;

vocabulary obtained from a database associated with the user;

vocabulary associated with application preferences;

vocabulary obtained from usage history; and

current dialog state.

30. The computer program product of claim 25, wherein the computer program code configured to cause at least one processor to prompt the user comprises computer program code configured to cause at least one processor to generate at least one prompt based at least in part on the received context information.

31. The computer program product of claim 25, wherein the computer program code configured to cause at least one processor to interpret the received user input to derive a representation of user intent comprises computer program code configured to cause at least one processor to perform natural language processing on the received user input based at least in part on the received context information.

32. The computer program product of claim 31, wherein the received context information used in interpreting the received user input comprises at least one selected from the group consisting of:

data describing an event;

application context;

input previously provided by the user;

known information about the user;

location;

date;

environmental conditions; and

history.

33. The computer program product of claim 25, wherein the computer program code configured to cause at least one processor to identify at least one task and at least one parameter for the task comprises computer program code configured to cause at least one processor to identify at least one task and at least one parameter for the task based at least in part on the received context information.

34. The computer program product of claim 33, wherein the received context information used in identifying at least one task and at least one parameter for the task comprises at least one selected from the group consisting of:

data describing an event;

data from a database associated with the user;

data received from at least one sensor;

application context;

input previously provided by the user;

known information about the user;

location;

date;

environmental conditions; and

history.

35. The computer program product of claim 25, wherein the computer program code configured to cause at least one processor to generate a dialog response comprises computer program code configured to cause at least one processor to generating a dialog response based at least in part on the received context information.

36. The computer program product of claim 35, wherein the received context information used in generating a dialog response comprises at least one selected from the group consisting of:

data from a database associated with the user;

application context;

input previously provided by the user;

known information about the user;

location;

date;

environmental conditions; and

history.

37. The computer program product of claim 25, wherein the computer program code configured to cause at least one processor to receive context information from a context source comprises:

computer program code configured to cause at least one processor to request the context information from a context source; and

computer program code configured to cause at least one processor to receive the context information in response to the request.

38. The computer program product of claim 25, wherein the computer program code configured to cause at least one processor to receive context information from a context source comprises:

computer program code configured to cause at least one processor to receive at least a portion of the context information prior to receiving user input.

39. The computer program product of claim 25, wherein the computer program code configured to cause at least one processor to receive context information from a context source comprises:

computer program code configured to cause at least one processor to receive at least a portion of the context information after receiving user input.

40. The computer program product of claim 25, wherein the computer program code configured to cause at least one processor to receive context information from a context source comprises:

computer program code configured to cause at least one processor to receive static context information as part of an initialization step; and

the computer program code configured to cause at least one processor to receive additional context information after receiving user input.

41. The computer program product of claim 25, wherein the computer program code configured to cause at least one processor to receive context information from a context source comprises:

computer program code configured to cause at least one processor to receive push notification of a change in context information; and

computer program code configured to cause at least one processor to, responsive to the push notification, update locally stored context information.

42. A system for interpreting user input to perform a task, comprising:

an output device, configured to prompt a user for input;

an input device, configured to receive user input;

at least one processor, communicatively coupled to the output device and to the input device, configured to perform the steps of;

receiving context information from a context source;

interpreting the received user input to derive a representation of user intent;

identifying at least one task and at least one parameter for the task, based at least in part on the derived representation of user intent;

executing the at least one task using the at least one parameter, to derive a result; and

generating a dialog response based on the derived result;

wherein the output device is further configured to output the generated dialog response;

and wherein at least one of prompting the user for input, interpreting the receiving user input, identifying the at least one task and at least one parameter for the task, and generating the dialog response is performed using the received context information.

43. The system of claim 42, wherein the received context information comprises at least one selected from the group consisting of:

application context;

personal data associated with the user;

data from a database associated with the user;

data obtained from dialog history;

data received from at least one sensor;

application preferences;

application usage history;

data describing an event;

data describing an acoustic environment in which the spoken input is received;

current dialog state;

input previously provided by the user;

location;

local time; and

environmental conditions.

44. The system of claim 42, wherein:

the output device is configured to prompt the user via a conversational interface; and

the input device is configured to receive user input via the conversational interface by receiving spoken input;

and wherein the at least one processor is configured to convert the spoken input to a text representation.

45. The system of claim 44, wherein the at least one processor is configured to convert the spoken input to a text representation by:

generating a plurality of candidate text interpretations of the spoken input; and

ranking at least a subset of the generated candidate text interpretations;

wherein at least one of the generating and ranking steps is performed using the received context information.

46. The system of claim 45, wherein the received context information used in at least one of the generating and ranking comprises at least one selected from the group consisting of:

data describing an acoustic environment in which the spoken input is received;

data received from at least one sensor;

vocabulary obtained from a database associated with the user;

vocabulary associated with application preferences;

vocabulary obtained from usage history; and

current dialog state.

47. The system of claim 42, wherein the output device is configured to prompt the user by generating at least one prompt based at least in part on the received context information.

48. The system of claim 42, wherein the at least one processor is configured to interpret the received user input to derive a representation of user intent by performing natural language processing on the received user input based at least in part on the received context information.

49. The system of claim 48, wherein the received context information used in interpreting the received user input comprises at least one selected from the group consisting of:

data describing an event;

application context;

input previously provided by the user;

known information about the user;

location;

date;

environmental conditions; and

history.

50. The system of claim 42, wherein the at least one processor is configured to identify at least one task and at least one parameter for the task by identifying at least one task and at least one parameter for the task based at least in part on the received context information.

51. The system of claim 50, wherein the received context information used in identifying at least one task and at least one parameter for the task comprises at least one selected from the group consisting of:

data describing an event;

data from a database associated with the user;

data received from at least one sensor;

application context;

input previously provided by the user;

known information about the user;

location;

date;

environmental conditions; and

history.

52. The system of claim 42, wherein the at least one processor is configured to generate a dialog response by generating a dialog response based at least in part on the received context information.

53. The system of claim 52, wherein the received context information used in generating a dialog response comprises at least one selected from the group consisting of:

data from a database associated with the user;

application context;

input previously provided by the user;

known information about the user;

location;

date;

environmental conditions; and

history.

54. The system of claim 42, wherein the received context information comprises at least one selected from the group consisting of:

context information stored at a server; and

context information stored at a client.

55. The system of claim 42, wherein the at least one processor is configured to receive context information from a context source by:

requesting the context information from a context source; and

receiving the context information in response to the request.

56. The system of claim 42, wherein the at least one processor is configured to receive context information from a context source by:

receiving at least a portion of the context information prior to receiving user input.

57. The system of claim 42, wherein the at least one processor is configured to receive context information from a context source by:

receiving at least a portion of the context information after receiving user input.

58. The system of claim 42, wherein the at least one processor is configured to receive context information from a context source by:

receiving static context information as part of an initialization step; and

receiving additional context information after receiving user input.

59. The system of claim 42, wherein the at least one processor is configured to receive context information from a context source by:

receiving push notification of a change in context information; and

responsive to the push notification, updating locally stored context information.

60. The system of claim 42, wherein the output device, input device, and at least one processor are implemented as components of at least one selected from the group consisting of:

a telephone;

a smartphone;

a tablet computer;

a laptop computer;

a personal digital assistant;

a desktop computer;

a kiosk;

a consumer electronic device;

a consumer entertainment device;

a music player;

a camera;

a television;

an electronic gaming unit; and

a set-top box.