Unconventional virtual assistant interactions

• US 10,956,666 B2
• Filed: 05/25/2016
• Issued: 03/23/2021
• Est. Priority Date: 11/09/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Vescovi, Marcos Regis, Vega Galvez, Tomas A., Karashchuk, Pierre, Gruber, Thomas R., Guzzoni, Didier R.
• Primary Examiner(s): Stork, Kyle R
• Application Number:
  US15/164,630
• Publication Number:
  US 20170132199A1
• Time in Patent Office: 1,763 Days
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/186 Templates
• CPC Class Codes: G06Q 10/06311 Scheduling, planning or task assignment for a person or group
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

An exemplary method of using a virtual assistant may include receiving an unstructured natural-language user request for a service from a virtual assistant; determining whether the user request corresponds to at least one of a plurality of plan templates accessible to the virtual assistant, based on the content of the user request and the content of the plan templates; in accordance with a determination that the user request corresponds to at least one of the plurality of plan templates, selecting one of the plurality of plan templates; and in accordance with a determination the user request fails to correspond to at least one of the plurality of plan templates, foregoing selection of a plan template; and in response to selection of one of the plan templates, beginning to respond to the user request according to the selected plan template.

Claims

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device, cause the electronic device to:

receive an unstructured natural-language user request for a service from a virtual assistant;

determine whether the user request corresponds to at least one of a plurality of plan templates accessible to the virtual assistant, based on the content of the user request and the content of the plan templates, wherein a plan template of the plurality of plan templates includes a set of sequential instructions and corresponding one or more inputs and outputs, and at least one of the one or more inputs is a contingent input that corresponds to a subsequent action;

in accordance with a determination that the user request corresponds to at least one of the plurality of plan templates;

select one of the plurality of plan templates;

determine whether sufficient information is available to the virtual assistant for the virtual assistant to generate a plan based on the selected plan template;

in accordance with a determination that sufficient information is available to the virtual assistant for the virtual assistant to generate the plan based on the selected plan template, generate the plan based on the selected plan template with the virtual assistant, wherein an execution of at least a portion of the plan is contingent upon receiving the contingent input at a time subsequent to generating the plan; and

in accordance with a determination that insufficient information is available to the virtual assistant for the virtual assistant to generate the plan based on the selected plan template;

obtain sufficient information to generate the plan based on the selected plan template; and

in response to sufficient information being obtained, generate the plan based on the selected plan template with the virtual assistant; and

in accordance with a determination that the user request fails to correspond to at least one of the plurality of plan templates, forego selection of a plan template;

in response to selection of one of the plan templates, begin to respond to the user request according to the selected plan template by causing execution of a first instruction of the set of sequential instructions; and

in response to receiving the contingent input, continue to respond to the user request by causing execution of a second instruction of the set of sequential instructions according to the subsequent action.

2. The non-transitory computer-readable storage medium of claim 1, wherein the instructions to cause the electronic device to obtain sufficient information to generate a plan based on the plan template further comprise instructions to cause the electronic device to:

initiate communication with the user to request sufficient information to generate a plan based on plan template; and

in response to the initiation, receive sufficient information to generate a plan based on the plan template.

3. The non-transitory computer-readable storage medium of claim 1, wherein the instructions to cause the electronic device to obtain sufficient information to generate a plan based on the plan template further comprise instructions to cause the electronic device to:

initiate communication with an entity other than the user to request sufficient information to generate a plan based on the plan template; and

in response to the initiation, receive sufficient information to generate a plan based on the plan template.

4. The non-transitory computer-readable storage medium of claim 1, wherein the instructions to cause the electronic device to obtain sufficient information to generate a plan based on the plan template further comprise instructions to cause the electronic device to:

initiate communication with an external service to request sufficient information to generate a plan based on the plan template; and

in response to the initiation;

receive sufficient information to generate a plan based on the plan template.

5. The non-transitory computer-readable storage medium of claim 1, wherein the instructions to cause the electronic device to obtain sufficient information to generate a plan based on the plan template further comprise instructions to cause the electronic device to:

obtain sufficient information from the electronic device to generate a plan based on the plan template.

6. The non-transitory computer-readable storage medium of claim 1, wherein the sufficient information from the electronic device to generate a plan based on the plan template includes a time associated with task execution.

7. The non-transitory computer-readable storage medium of claim 1, wherein the sufficient information from the electronic device to generate a plan based on the plan template includes a location of task execution.

8. The non-transitory computer-readable storage medium of claim 1, wherein the sufficient information from the electronic device to generate a plan based on the plan template includes an identity of at least one person associated with task execution.

9. The non-transitory computer-readable storage medium of claim 1, further comprising instructions to cause the electronic device to:

after receipt of the user request, determine whether the user request comprises a plan trigger;

in accordance with a determination that the user request comprises a plan trigger, initiate the determination whether sufficient information is available to the virtual assistant for the virtual assistant to generate a plan based on the selected plan template; and

in accordance with a determination that the user request does not comprise a plan trigger;

respond to the user request other than by generation of a plan based on at least one plan template.

10. The non-transitory computer-readable storage medium of claim 1, wherein the instructions to cause the electronic device to generate a plan based on the selected plan template further comprise instructions to cause the electronic device to:

generate the plan based in part on at least one of device context and user context.

11. The non-transitory computer-readable storage medium of claim 10, wherein the device context includes a location of the device.

12. The non-transitory computer-readable storage medium of claim 10, wherein the device context includes motion of the device.

13. The non-transitory computer-readable storage medium of claim 10, wherein the device context includes proximity to a second electronic device.

14. The non-transitory computer-readable storage medium of claim 10, wherein the device context includes weather conditions in proximity to the electronic device.

15. The non-transitory computer-readable storage medium of claim 10, wherein the user context includes the content of the user input.

16. The non-transitory computer-readable storage medium of claim 10, wherein the user context includes media associated with the user.

17. The non-transitory computer-readable storage medium of claim 10, wherein the user context includes demographic information associated with the user.

18. The non-transitory computer-readable storage medium of claim 10, wherein the user context includes data associated with user content accessible by the device.

19. The non-transitory computer-readable storage medium of claim 10, wherein the user context includes content of messages stored on the electronic device.

20. The non-transitory computer-readable storage medium of claim 10, wherein the user context includes contact information.

21. The non-transitory computer-readable storage medium of claim 10, wherein the user context includes at least one location associated with the user.

22. The non-transitory computer-readable storage medium of claim 10, wherein the user context includes biometric information.

23. The non-transitory computer-readable storage medium of claim 1, further comprising instructions to cause the electronic device to:

execute the generated plan.

24. The non-transitory computer-readable storage medium of claim 23, further comprising instructions to cause the electronic device to:

during execution of the plan, detect an exception to the plan; and

determine whether the exception can be handled according to the plan;

in accordance with a determination that the exception can be handled according to the plan, handle, with the virtual assistant, the exception with information associated with the plan; and

in accordance with a determination that the exception cannot he handled according to the plan;

obtain sufficient information for the virtual assistant to handle the exception; and

handle, with the virtual assistant, the exception with the obtained information.

25. The non-transitory computer-readable storage medium of claim 24, wherein the instructions to cause the electronic device to obtain sufficient information for the virtual assistant to handle the exception further comprise instructions to cause the electronic device to:

initiate communication with the user from whom the request for service was received to request sufficient information for the virtual assistant to handle the exception; and

in response to the initiation, receive sufficient information for the virtual assistant to handle the exception.

26. The non-transitory computer-readable storage medium of claim 24, wherein the instructions to cause the electronic device to obtain sufficient information for the virtual assistant to handle the exception further comprise instructions to cause the electronic device to:

initiate communication with an entity other than the user to request sufficient information for the virtual assistant to handle the exception; and

in response to the initiation, receive sufficient information for the virtual assistant to handle the exception.

27. The non-transitory computer-readable storage medium of claim 24, wherein the instructions to cause the electronic device to obtain sufficient information for the virtual assistant to handle the exception further comprise instructions to cause the electronic device to:

initiate communication with an external service to request sufficient information for the virtual assistant to handle the exception; and

in response to the initiation, receive sufficient information for the virtual assistant to handle the exception.

28. The non-transitory computer-readable storage medium of claim 24, wherein the instructions to cause the electronic device to execute the plan further comprise instructions to cause the electronic device to:

monitor execution of the plan over time with the virtual assistant for receipt of at least one input upon which execution of the plan is contingent;

receive the at least one input; and

continue execution of the plan.

29. The non-transitory computer-readable storage medium of claim 28, wherein at least one input includes multimodal input.

30. The non-transitory computer-readable storage medium of claim 24, wherein the instructions to cause the electronic device to execute the plan further comprise instructions to cause the electronic device to:

cause at least one physical output.

31. The non-transitory computer-readable storage medium of claim 24, wherein the instructions to cause the electronic device to execute the plan further comprise instructions to cause the electronic device to:

generate multimodal output.

32. The non-transitory computer-readable storage medium of claim 24, wherein the instructions to cause the electronic device to execute the plan further comprise instructions to cause the electronic device to:

initiate transmission of at least one message.

33. The non-transitory computer-readable storage medium of claim 24, further comprising instructions to cause the electronic device to:

cause presentation of output to an entity other than the user.

34. The non-transitory computer-readable storage medium of claim 24, further comprising instructions to cause the electronic device to:

notify the user after execution of the plan is complete.

35. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs comprise further instructions to cause the electronic device to:

determine whether additional information is required for the virtual assistant to generate a plan for responding to the user request based on the selected plan template;

in accordance with a determination that additional information is required,

query the user based on at least one of the set of instructions from the plan template, and

receive, from the user, at least one of the one or more inputs.

36. A method of using a virtual assistant, comprising:

at an electronic device configured to transmit and receive data,

receiving an unstructured natural-language user request for a service from a virtual assistant;

determining whether the user request corresponds to at least one of a plurality of plan templates accessible to the virtual assistant, based on the content of the user request and the content of the plan templates, wherein a plan template of the plurality of plan templates includes a set of sequential instructions and corresponding one or more inputs and outputs, and at least one of the one or more inputs is a contingent input that corresponds to a subsequent action;

in accordance with a determination that the user request corresponds to at least one of the plurality of plan templates;

selecting one of the plurality of plan templates;

determining whether sufficient information is available to the virtual assistant for the virtual assistant to generate a plan based on the selected plan template;

in accordance with a determination that sufficient information is available to the virtual assistant for the virtual assistant to generate the plan based on the selected plan template, generating the plan based on the selected plan template with the virtual assistant; and

in accordance with a determination that insufficient information is available to the virtual assistant for the virtual assistant to generate the plan based on the selected plan template, wherein an execution of at least a portion of the plan is contingent upon receiving the contingent input at a time subsequent to generating the plan;

obtain sufficient information to generate the plan based on the selected plan template; and

in response to sufficient information being obtained, generating the plan based on the selected plan template with the virtual assistant; and

in accordance with a determination the user request fails to correspond to at least one of the plurality of plan templates, foregoing selection of a plan template;

in response to selection of one of the plan templates, beginning to respond to the user request according to the selected plan template by causing execution of a first instruction of the set of sequential instructions; and

in response to receiving the contingent input, continue to respond to the user request by causing execution of a second instruction of the set of sequential instructions according to the subsequent action.

37. An electronic device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving an unstructured natural-language user request for a service from a virtual assistant;

determining whether the user request corresponds to at least one of a plurality of plan templates accessible to the virtual assistant, based on the content of the user request and the content of the plan templates, wherein a plan template of the plurality of plan templates includes a set of sequential instructions and corresponding one or more inputs and outputs, and at least one of the one or more inputs is a contingent input that corresponds to a subsequent action;

in accordance with a determination that the user request corresponds to at least one of the plurality of plan templates;

selecting one of the plurality of plan templates;

determining whether sufficient information is available to the virtual assistant for the virtual assistant to generate a plan based on the selected plan template;

in accordance with a determination that sufficient information is available to the virtual assistant for the virtual assistant to generate the plan based on the selected plan template, generating the plan based on the selected plan template with the virtual assistant; and

in accordance with a determination that insufficient information is available to the virtual assistant for the virtual assistant to generate the plan based on the selected plan template, wherein an execution of at least a portion of the plan is contingent upon receiving the contingent input at a time subsequent to generating the plan;

obtain sufficient information to generate the plan based on the selected plan template; and

in response to sufficient information being obtained, generating the plan based on the selected plan template with the virtual assistant; and

in accordance with a determination the user request fails to correspond to at least one of the plurality of plan templates, foregoing selection of a plan template;

in response to selection of one of the plan templates, beginning to respond to the user request according to the selected plan template by causing execution of a first instruction of the set of sequential instructions; and

in response to receiving the contingent input, continue to respond to the user request by causing execution of a second instruction of the set of sequential instructions according to the subsequent action.