Intelligent automated assistant

• US 10,553,215 B2
• Filed: 07/02/2018
• Issued: 02/04/2020
• Est. Priority Date: 09/23/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Saddler, Harry J., Piercy, Aimee T., Weinberg, Garrett L., Booker, Susan L.
• Primary Examiner(s): Godbold, Douglas
• Application Number:
  US16/025,450
• Publication Number:
  US 20180308486A1
• Time in Patent Office: 582 Days
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 13/02 Methods for producing synthetic speech; Speech synthesisers
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/221 Announcement of recognition results
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech

Abstract

Systems and processes for operating an automated assistant are disclosed. In one example process, an electronic device provides an audio output via a speaker of the electronic device. While providing the audio output, the electronic device receives, via a microphone of the electronic device, a natural language speech input. The electronic device derives a representation of user intent based on the natural language speech input and the audio output, identifies a task based on the derived user intent; and performs the identified task.

Claims

1. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the device to:

sample, via a microphone of the electronic device, an audio input;

wherein the audio input comprises a spoken trigger;

in response to sampling the audio input, provide, via a speaker of the electronic device, an audio output indicative of a state of an automated assistant;

while providing the audio output indicative of a state of the automated assistant;

receive a natural language speech input;

derive a representation of user intent based on the natural language speech input;

perform a task based on the user intent; and

output, via a speaker of the electronic device, one or more results based on the performed task;

after outputting the one or more results, cease to provide the audio output;

within a predetermined duration after ceasing to provide the audio output, receive a second audio input, wherein the second audio input does not comprise the spoken trigger;

derive a representation of another user intent based on the second audio input; and

perform another task based on the another user intent.

2. The non-transitory computer readable storage medium of claim 1,

wherein ceasing to provide the audio output includes ceasing to provide the audio output after passage of a predefined period of time after outputting the one or more results based on the performed task.

3. The non-transitory computer readable storage medium of claim 1, wherein the audio output is indicative of an active state of the automated assistant.

4. The non-transitory computer readable storage medium of claim 1, wherein the audio output is a continuous sound.

5. The non-transitory computer readable storage medium of claim 1, wherein the audio output comprises white noise.

6. The non-transitory computer readable storage medium of claim 1, wherein the audio output comprises sound of running water.

7. The non-transitory computer readable storage medium of claim 1, wherein providing the audio output comprises attenuating the audio output over a period of time.

9. The non-transitory computer readable storage medium of claim 1, wherein the audio output indicative of a state of the automated assistant is a first audio output, and wherein the one or more programs further comprise instructions, which when executed by one or more processors of the electronic device, cause the device to:

while providing the first audio output, provide a third audio output indicative of a completed processing of a user request.

10. The non-transitory computer readable storage medium of claim 1, wherein the audio output indicative of a state of the automated assistant is a first audio output, and wherein the one or more programs further comprise instructions, which when executed by one or more processors of the electronic device, cause the device to:

while providing the first audio output, provide a fourth audio output indicative of an acknowledgement of the spoken trigger.

11. The non-transitory computer readable storage medium of claim 1, wherein the audio output indicative of a state of the automated assistant is a first audio output, and wherein the one or more programs further comprise instructions, which when executed by one or more processors of the electronic device, cause the device to:

while providing the first audio output indicative of a state of the automated assistant, provide a fifth audio output indicative of an error.

12. The non-transitory computer readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by one or more processors of the electronic device, cause the device to:

determine a user context; and

wherein deriving the representation of the another user intent includes deriving the representation of the another user intent based on the user context.

13. The non-transitory computer readable storage medium of claim 12, wherein the user context is based on the natural language speech input, the user intent, the task, or a combination thereof.

14. The non-transitory computer readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by one or more processors of the electronic device, cause the device to:

after sampling, via the microphone of the electronic device, the audio input, determine a current context; and

adjust the audio output based on the determined current context.

15. The non-transitory computer readable storage medium of claim 14, wherein the current context comprises a current time, and wherein adjusting the audio output comprises adjusting a volume of the audio output.

16. The non-transitory computer readable storage medium of claim 1, wherein the audio output indicative of a state of the automated assistant is based on a type of the electronic device.

8. The non-transitory computer readable storage medium of claim wherein the audio output indicative of a state of the automated assistant is a first audio output, and wherein the one or more programs further comprise instructions, which when executed by one or more processors of the electronic device, cause the device to:

while providing the first audio output, provide a second audio output indicative of an ongoing processing of a user request.

17. A method for operating an automated assistant, comprising:

at an electronic device with a microphone and a speaker,

sampling, via the microphone of the electronic device, an audio input, wherein the audio input comprises a spoken trigger;

in response to sampling the audio input, providing, via the speaker of the electronic device, an audio output indicative of a state of the automated assistant;

while providing the audio output indicative of a state of the automated assistant;

receiving a natural language speech input;

deriving a representation of user intent based on the natural language speech input;

performing a task based on the user intent; and

outputting, via the speaker, one or more results based on the performed task;

after outputting the one or more results, ceasing to provide the audio output;

within a predetermined duration after ceasing to provide the audio output, receiving a second audio input, wherein the second audio input does not comprise the spoken trigger;

deriving a representation of another user intent based on the second audio input; and

performing another task based on the another user intent.

18. The method of claim 17, wherein ceasing to provide the audio output includes ceasing to provide the audio output after passage of a predefined period of time after outputting the one or more results based on the performed task.

19. The method of claim 17, wherein the audio output is indicative of an active state of the automated assistant.

20. The method of claim 17, wherein the audio output is a continuous sound.

21. The method of claim 17, wherein the audio output comprises white noise.

22. The method of claim 17, wherein the audio output comprises sound of running water.

23. The method of claim 17, wherein providing the audio output comprises attenuating the audio output over a period of time.

24. The method of claim 17, wherein the audio output indicative of a state of the automated assistant is a first audio output, the method further comprising:

while providing the first audio output, providing a second audio output indicative of an ongoing processing of a user request.

25. The method of claim 17, wherein the audio output indicative of a state of the automated assistant is a first audio output, the method further comprising:

while providing the first audio output, providing a third audio output indicative of a completed processing of a user request.

26. The method of claim 17, wherein the audio output indicative of a state of the automated assistant is a first audio output, the method further comprising:

while providing the first audio output, providing a fourth audio output indicative of an acknowledgement of the spoken trigger.

27. The method of claim 17, wherein the audio output indicative of a state of the automated assistant is a first audio output, the method further comprising:

while providing the first audio output indicative of a state of the automated assistant, providing a fifth audio output indicative of an error.

28. The method of claim 17, further comprising:

determining a user context; and

wherein deriving the representation of the another user intent includes deriving the representation of the another user intent based on the user context.

29. The method of claim 28, wherein the user context is based on the natural language speech input, the user intent, the task, or a combination thereof.

30. The method of claim 17, further comprising:

after sampling, via the microphone of the electronic device, the audio input, determining a current context; and

adjusting the audio output based on the determined current context.

31. The method of claim 30, wherein the current context comprises a current time, and wherein adjusting the audio output comprises adjusting a volume of the audio output.

32. The method of claim 17, wherein the audio output indicative of a state of the automated assistant is based on a type of the electronic device.

33. An electronic device for operating an automated assistant, the electronic device comprising:

one or more processors;

a memory;

a microphone;

a speaker; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

sampling, via the microphone of the electronic device, an audio input, wherein the audio input comprises a spoken trigger;

in response to sampling the audio input, providing, via the speaker of the electronic device, an audio output indicative of a state of the automated assistant;

while providing the audio output indicative of a state of the automated assistant;

receiving a natural language speech input;

deriving a representation of user intent based on the natural language speech input;

performing a task based on the user intent; and

outputting, via the speaker, one or more results based on the performed task;

after outputting the one or more results, ceasing to provide the audio output;

within a predetermined duration after ceasing to provide the audio output, receiving a second audio input, wherein the second audio input does not comprise the spoken trigger;

deriving a representation of another user intent based on the second audio input; and

performing another task based on the another user intent.

34. The electronic device of claim 33, wherein ceasing to provide the audio output includes ceasing to provide the audio output after passage of a predefined period of time after outputting the one or more results based on the performed task.

35. The electronic device of claim 33, wherein the audio output is indicative of an active state of the automated assistant.

36. The electronic device of claim 33, wherein the audio output is a continuous sound.

37. The electronic device of claim 33, wherein the audio output comprises white noise.

38. The electronic device of claim 33, wherein the audio output comprises sound of running water.

39. The electronic device of claim 33, wherein providing the audio output comprises attenuating the audio output over a period of time.

40. The electronic device of claim 33, wherein the audio output indicative of a state of the automated assistant is a first audio output, and wherein the one or more programs further include instructions for:

while providing the first audio output, providing a second audio output indicative of an ongoing processing of a user request.

41. The electronic device of claim 33, wherein the audio output indicative of a state of the automated assistant is a first audio output, and wherein the one or more programs further include instructions for:

while providing the first audio output, providing a third audio output indicative of a completed processing of a user request.

42. The electronic device of claim 33, wherein the audio output indicative of a state of the automated assistant is a first audio output, and wherein the one or more programs further include instructions for:

while providing the first audio output, providing a fourth audio output indicative of an acknowledgement of the spoken trigger.

43. The electronic device of claim 33, wherein the audio output indicative of a state of the automated assistant is a first audio output, and wherein the one or more programs further include instructions for:

while providing the first audio output indicative of a state of the automated assistant, providing a fifth audio output indicative of an error.

44. The electronic device of claim 33, wherein the one or more programs further include instructions for:

determining a user context; and

wherein deriving the representation of the another user intent includes deriving the representation of the another user intent based on the user context.

45. The electronic device of claim 44, wherein the user context s based on the natural language speech input, the user intent, the task, or a combination thereof.

46. The electronic device of claim 33, wherein the one or more programs further include instructions for:

after sampling, via the microphone of the electronic device, the audio input, determining a current context; and

adjusting the audio output based on the determined current context.

47. The electronic device of claim 46, wherein the current context comprises a current time, and wherein adjusting the audio output comprises adjusting a volume of the audio output.

48. The electronic device of claim 33, wherein the audio output indicative of a state of the automated assistant is based on a type of the electronic device.