Virtual assistant activation

• US 10,529,332 B2
• Filed: 01/04/2018
• Issued: 01/07/2020
• Est. Priority Date: 03/08/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Lemay, Stephen O., Newendorp, Brandon J., Dascola, Jonathan R.
• Primary Examiner(s): Godbold, Douglas
• Application Number:
  US15/862,286
• Publication Number:
  US 20180130470A1
• Time in Patent Office: 733 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/28 Constructional details of speech recognition systems
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/10 Multimedia information

Abstract

At an electronic device with a display, a microphone, and an input device: while the display is on, receiving user input via the input device, the user input meeting a predetermined condition; in accordance with receiving the user input meeting the predetermined condition, sampling audio input received via the microphone; determining whether the audio input comprises a spoken trigger; and in accordance with a determination that audio input comprises the spoken trigger, triggering a virtual assistant session.

Claims

1. An electronic device comprising:

a display;

one or more processors;

a memory;

a microphone; and

one or more programs, wherein the one or more programs are stored in memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

detecting a software event meeting a predetermined condition;

in accordance with a determination that the software event meeting the predetermined condition is detected, sampling audio input received via the microphone;

determining whether audio input received via the microphone comprises a spoken trigger; and

in accordance with a determination that the audio input comprises the spoken trigger, triggering a virtual assistant session;

while sampling audio input, receiving a signal associated with a user lowering the electronic device;

determining whether an amount of lowering of the electronic device during a time interval exceeds a predetermined threshold; and

in accordance with a determination that the amount of lowering of the electronic device during the time interval exceeds the predetermined threshold, ceasing the sampling of audio input.

2. The electronic device of claim 1, wherein the predetermined condition includes an event triggered by an application of the electronic device.

3. The electronic device of claim 1, wherein the predetermined condition comprises an event representing a notification.

4. The electronic device of claim 1, wherein the predetermined condition comprises an event representing an incoming calendar invitation from an external device.

5. The electronic device of claim 1, wherein the predetermined condition comprises an event representing an incoming transmission, the transmission triggered from an external device.

6. The electronic device of claim 5, wherein the incoming transmission includes an incoming phone call from the external device.

7. The electronic device of claim 5, wherein the incoming transmission includes an incoming text or multimedia message from the external device.

8. The electronic device of claim 4, the one or more programs further including instructions for:

in accordance with triggering a virtual assistant session, initiating a response addressed to the external device.

9. The electronic device of claim 1, the one or more programs further including instructions for:

determining whether the display is on, and

wherein the sampling of audio input occurs in accordance with a determination that the display is on.

10. The electronic device of claim 1, the one or more programs further including instructions for:

ceasing the sampling of audio input after the sampling of audio input has occurred for a predetermined duration of time.

11. The electronic device of claim 1, the one or more programs further including instructions for:

while sampling audio input in accordance with the determination that the software event meeting the condition is detected, receiving a user input meeting a second predetermined condition; and

in accordance with receiving the user input meeting the second predetermined condition, ceasing the sampling of audio input.

12. The electronic device of claim 11, wherein the second predetermined condition comprises detecting activation of a button of the electronic device.

13. The electronic device of claim 11, wherein the second predetermined condition comprises detecting touch input on a touch-sensitive surface of the electronic device.

14. The electronic device of claim 1, the one or more programs further including instructions for:

while sampling audio input in accordance with the determination that the software event meeting the condition is detected, determining whether the display is off; and

in accordance with a determination that the display is off, ceasing the sampling of audio input.

15. The electronic device of claim 1, the one or more programs further including instructions for:

while sampling audio input in accordance with the determination that the software event meeting the condition is detected, identifying an audio endpoint in the audio input; and

in response to identifying the audio endpoint, ceasing the sampling of audio input.

16. The electronic device of claim 10, the one or more programs further including instructions for:

in accordance with ceasing to sample audio input, providing a haptic output.

17. The electronic device of claim 1, the one or more programs further including instructions for:

in accordance with triggering the virtual assistant session, prompting a user for spoken input.

18. The electronic device of claim 1, wherein the audio input comprises the spoken trigger and additional input, the one or more programs further including instructions for:

in accordance with triggering the virtual assistant session;

determining a user intent based on at least the additional input in the audio input; and

executing a task associated with the user intent.

19. The electronic device of claim 1, the one or more programs further including instructions for:

in accordance with a determination that the electronic device is operating in a predetermined mode, forgoing sampling of audio input, even after detecting the software event meeting the predetermined condition,

wherein operating in the predetermined mode comprises muting audio output of the electronic device.

20. The electronic device of claim 1, wherein the spoken trigger comprises a predetermined phrase.

21. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device with a display and a microphone, cause the electronic device to:

detect a software event meeting a predetermined condition;

in accordance with a determination that the software event meeting the predetermined condition is detected, sample audio input received via the microphone;

determine whether audio input received via the microphone comprises a spoken trigger; and

in accordance with a determination that the audio input comprises the spoken trigger, trigger a virtual assistant session;

while sampling audio input, receive a signal associated with a user lowering the electronic device;

determine whether an amount of lowering of the electronic device during a time interval exceeds a predetermined threshold; and

in accordance with a determination that the amount of lowering of the electronic device during the time interval exceeds a predetermined threshold, cease the sampling of audio input.

22. The non-transitory computer readable storage medium of claim 21, wherein the predetermined condition includes an event triggered by an application of the electronic device.

23. The non-transitory computer readable storage medium of claim 21, wherein the predetermined condition comprises an event representing an incoming transmission, the transmission triggered from an external device.

24. The non-transitory computer readable storage medium of claim 23, the one or more programs further comprising instructions, which when executed by an electronic device with a display and a microphone cause the electronic device to:

in accordance with triggering a virtual assistant session, initiate a response addressed to the external device.

25. The non-transitory computer readable storage medium of claim 21, the one or more programs further comprising instructions, which when executed by an electronic device with a display and a microphone, cause the electronic device to:

determine whether the display is on, and

wherein the sampling of audio input occurs in accordance with a determination that the display is on.

26. The non-transitory computer readable storage medium of claim 21, the one or more programs further comprising instructions, which when executed by an electronic device with a display and a microphone, cause the electronic device to:

cease the sampling of audio input after the sampling of audio input has occurred for a predetermined duration of time.

27. The non-transitory computer readable storage medium of claim 21, the one or more programs further comprising instructions, which when executed by an electronic device with a display and a microphone, cause the electronic device to:

while sampling audio input in accordance with the determination that the software event meeting the condition is detected, receive a user input meeting a second predetermined condition; and

in accordance with receiving the user input meeting the second predetermined condition, cease the sampling of audio input.

28. The non-transitory computer readable storage medium of claim 21, the one or more programs further comprising instructions, which when executed by an electronic device with a display and a microphone, cause the electronic device to:

while sampling audio input in accordance with the determination that the software event meeting the condition is detected, determine whether the display is off; and

in accordance with a determination that the display is off, cease the sampling of audio input.

29. The non-transitory computer readable storage medium of claim 21, the one or more programs further comprising instructions, which when executed by an electronic device with a display and a microphone, cause the electronic device to:

while sampling audio input in accordance with the determination that the software event meeting the condition is detected, identify an audio endpoint in the audio input; and

in response to identifying the audio endpoint, cease the sampling of audio input.

30. The non-transitory computer readable storage medium of claim 21, wherein the audio input comprises the spoken trigger and additional input, the one or more programs further comprising instructions, which when executed by an electronic device with a display and a microphone, cause the electronic device to:

in accordance with triggering the virtual assistant session;

determine a user intent based on at least the additional input in the audio input, and

execute a task associated with the user intent.

31. The non-transitory computer readable storage medium of claim 21, the one or more programs further comprising instructions, which when executed by an electronic device with a display and a microphone, cause the electronic device to:

in accordance with a determination that the electronic device is operating in a predetermined mode, forgo sampling of audio input, even after detecting the software event meeting the predetermined condition,

wherein operating in the predetermined mode comprises muting audio output of the electronic device.

32. A method, comprising:

at an electronic device with a display and a microphone;

detecting a software event meeting a predetermined condition;

in accordance with a determination that the software event meeting the predetermined condition is detected, sampling audio input received via a microphone;

determining whether audio input received via the microphone comprises a spoken trigger; and

in accordance with a determination that the audio input comprises the spoken trigger, triggering a virtual assistant session;

while sampling audio input, receiving a signal associated with a user lowering the electronic device;

determining whether an amount of lowering of the electronic device during a time interval exceeds a predetermined threshold; and

in accordance with a determination that the amount of lowering of the electronic device during the time interval exceeds a predetermined threshold, ceasing the sampling of audio input.

33. The method of claim 32, wherein the predetermined condition is that the software event is an event triggered by an application of the electronic device.

34. The method of claim 32, wherein the predetermined condition comprises the software event representing an incoming transmission, the transmission triggered from an external device.

35. The method of claim 34, further comprising:

in accordance with triggering a virtual assistant session, initiating a response addressed to the external device.

36. The method of claim 32, further comprising:

determining whether the display is on, and

wherein the sampling of audio input occurs in accordance with a determination that the display is on.

37. The method of claim 32, further comprising:

ceasing the sampling of audio input after the sampling of audio input has occurred for a predetermined duration of time.

38. The method of claim 32, further comprising:

while sampling audio input in accordance with the determination that the software event meeting the condition is detected, receiving a user input meeting a second predetermined condition; and

in accordance with receiving the user input meeting the second predetermined condition, ceasing the sampling of audio input.

39. The method of claim 32, further comprising:

while sampling audio input in accordance with the determination that the software event meeting the condition is detected, determining whether the display is off; and

in accordance with a determination that the display is off, ceasing the sampling of audio input.

40. The method of claim 32, further comprising:

while sampling audio input in accordance with the determination that the software event meeting the condition is detected, identifying an audio endpoint in the audio input; and

in response to identifying the audio endpoint, ceasing the sampling of audio input.

41. The method of claim 32, wherein the audio input comprises the spoken trigger and additional input, further comprising:

in accordance with triggering the virtual assistant session;

determining a user intent based on at least the additional input in the audio input; and

executing a task associated with the user intent.

42. The method of claim 32, further comprising:

in accordance with a determination that the electronic device is operating in a predetermined mode, forgoing sampling of audio input, even after detecting the software event meeting the predetermined condition,

wherein operation in the predetermined mode comprises muting audio output of the electronic device.