Zero latency digital assistant

• US 11,126,400 B2
• Filed: 06/23/2020
• Issued: 09/21/2021
• Est. Priority Date: 09/08/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Stasior, William F., Carson, David A., Dasari, Rohit, Kim, Yoon
• Primary Examiner(s): To, Jennifer N
• Assistant Examiner(s): Fortino, Ashley M
• Application Number:
  US16/909,852
• Publication Number:
  US 20200319850A1
• Time in Patent Office: 455 Days
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/0673 Single storage device
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/285 Memory allocation or algorithm optimisation to reduce hardware requirements
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

An electronic device can implement a zero-latency digital assistant by capturing audio input from a microphone and using a first processor to write audio data representing the captured audio input to a memory buffer. In response to detecting a user input while capturing the audio input, the device can determine whether the user input meets a predetermined criteria. If the user input meets the criteria, the device can use a second processor to identify and execute a task based on at least a portion of the contents of the memory buffer.

Claims

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a first processor and a second processor of an electronic device with a microphone, cause the device to:

capture first audio input from the microphone, wherein the audio input includes speech from a user of the electronic device;

write, using the first processor, data representing the first audio input to a memory buffer, wherein the data includes at least a first portion of data representing at least a portion of the speech from a user of the electronic device;

while capturing audio input using the microphone and after writing the first portion of data to the memory buffer, receive a user input, wherein the user input is distinct from the first portion of audio data;

determine, using the first processor, whether the user input meets predetermined criteria;

in accordance with a determination that the user input meets the criteria, identify, using the second processor, a computing task based on at least the first portion of data wherein identifying the computing task includes identifying, based on at least the first portion of data, a user intent corresponding to the computing task; and

in accordance with a determination that the user input has not met the criteria, forgo identifying the task.

2. The non-transitory computer-readable storage medium according to claim 1, wherein identifying the computing task is further based on a second portion of data written to the buffer during detection of the user input.

3. The non-transitory computer-readable storage medium according to claim 1, further comprising instructions to cause the device to:

further in accordance with the determination that the user input meets the criteria, execute the identified computing task.

4. The non-transitory computer-readable storage medium according to claim 1, wherein identifying the computing task includes launching a digital assistant session on the second processor.

5. The non-transitory computer-readable storage medium according to claim 4, wherein the digital assistant session identifies and executes the computing task.

6. The non-transitory computer-readable storage medium according to claim 4, wherein launching the digital assistant session includes displaying a user interface associated with the digital assistant session.

7. The non-transitory computer-readable storage medium according to claim 6, wherein the user interface associated with the digital assistant session is displayed in a full-screen view.

8. The non-transitory computer-readable storage medium according to claim 4, wherein launching the digital assistant session includes activating one or more audio components on the device.

9. The non-transitory computer-readable storage medium according to claim 4, wherein the portion of the contents of the memory buffer is provided to a remote server associated with the digital assistant session.

10. The non-transitory computer-readable storage medium according to claim 4, further comprising instructions to cause the device to:

further in accordance with the determination that the user input meets the criteria;

activate a second microphone on the device, and

stream audio detected by the second microphone to the digital assistant session.

11. The non-transitory computer-readable storage medium according to claim 4, wherein the user input is a user input invoking the digital assistant session.

12. The non-transitory computer-readable storage medium according to claim 1, wherein the user input is a button press.

13. The non-transitory computer-readable storage medium according to claim 12, wherein the criteria includes a criterion that is met when the button press exceeds a predetermined threshold duration.

14. The non-transitory computer-readable storage medium according to claim 1, wherein the user input is audio data captured by the microphone and written to the buffer.

15. The non-transitory computer-readable storage medium according to claim 14, wherein the criteria includes a criterion that is met when the device determines that at least a second portion of the memory buffer includes audio data representing a predetermined trigger.

16. The non-transitory computer-readable storage medium according to claim 14, wherein the criteria includes a criterion that is met when the device determines that the audio data corresponds to an authorized user.

17. The non-transitory computer-readable storage medium according to claim 1, wherein the audio input is captured while the second processor is in a low-power mode, further comprising instructions to cause the device to:

further in accordance with the determination that the user input meets the criteria, cause the second processor to exit the low-power mode.

18. The non-transitory computer-readable storage medium according to claim 1, wherein the first processor is a processor of a first type and the second processor is a processor of a second type.

19. The non-transitory computer-readable storage medium according to claim 1, wherein identifying the computing task further includes identifying one or more parameters corresponding to the user intent.

20. A method, comprising:

at an electronic device including a microphone, a first processor, and a second processor;

capturing first audio input from the microphone, wherein the audio input includes speech from a user of the electronic device;

writing, using the first processor, data representing the first audio input to a memory buffer, wherein the data includes at least a first portion of data representing at least a portion of the speech from a user of the electronic device;

while capturing the audio input using the microphone and after writing the first portion of data to the memory buffer, receive a user input, wherein the user input is distinct from the first portion of audio data;

determining, using the first processor, whether the user input meets predetermined criteria;

in accordance with a determination that the user input meets the criteria, identifying, using the second processor, a computing task based on at least the first portion of data, wherein identifying the computing task includes identifying, based on at least the first portion of data, a user intent corresponding to the computing task; and

in accordance with a determination that the user input has not met the criteria, forgoing identifying the task.

21. The method of claim 20, wherein identifying the computing task is further based on a second portion of data written to the buffer during detection of the user input.

22. The method of claim 20, further comprising:

further in accordance with the determination that the user input meets the criteria, executing the identified computing task.

23. The method of claim 20, wherein identifying the computing task includes launching a digital assistant session on the second processor.

24. The method of claim 23, wherein the digital assistant session identifies and executes the computing task.

25. The method of claim 23, wherein launching the digital assistant session includes displaying a user interface associated with the digital assistant session.

26. The method of claim 25, wherein the user interface associated with the digital assistant session is displayed in a full-screen view.

27. The method of claim 23, wherein launching the digital assistant session includes activating one or more audio components on the device.

28. The method of claim 23, wherein the portion of the contents of the memory buffer is provided to a remote server associated with the digital assistant session.

29. The method of claim 23, further comprising:

further in accordance with the determination that the user input meets the criteria;

activating a second microphone on the device, and

streaming audio detected by the second microphone to the digital assistant session.

30. The method of claim 23, wherein the user input is a user input invoking the digital assistant session.

31. The method of claim 20, wherein the user input is a button press.

32. The method of claim 31, wherein the criteria includes a criterion that is met when the button press exceeds a predetermined threshold duration.

33. The method of claim 20, wherein the user input is audio data captured by the microphone and written to the buffer.

34. The method of claim 33, wherein the criteria includes a criterion that is met when the device determines that at least a second portion of the memory buffer includes audio data representing a predetermined trigger.

35. The method of claim 33, wherein the criteria includes a criterion that is met when the device determines that the audio data corresponds to an authorized user.

36. The method of claim 20, wherein the audio input is captured while the second processor is in a low-power mode, and further comprising:

further in accordance with the determination that the user input meets the criteria, causing the second processor to exit the low-power mode.

37. The method of claim 20, wherein the first processor is a processor of a first type and the second processor is a processor of a second type.

38. The method of claim 20, wherein identifying the computing task further includes identifying one or more parameters corresponding to the user intent.

39. An electronic device, comprising:

a microphone;

two or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the two or more processors, the one or more programs including instructions for;

capturing first audio input from the microphone, wherein the audio input includes speech from a user of the electronic device;

writing, using a first processor, data representing the first audio input to a memory buffer in the memory, wherein the data includes at least a first portion of data representing at least a portion of the speech from a user of the electronic device;

while capturing audio input using the microphone and after writing the first portion of data to the memory buffer, receive a user input, wherein the user input is distinct from the first portion of audio data;

determining, using the first processor, whether the user input meets predetermined criteria;

in accordance with a determination that the user input meets the criteria, identifying, using a second processor, a computing task based on at least the first portion of data, wherein identifying the computing task includes identifying, based on at least the first portion of data, a user intent corresponding to the computing task; and

in accordance with a determination that the user input has not met the criteria, forgoing identifying the task.

40. The electronic device of claim 39, wherein identifying the computing task is further based on a second portion of data written to the buffer during detection of the user input.

41. The electronic device of claim 39, the one or more programs further including instructions for:

further in accordance with the determination that the user input meets the criteria, executing the identified computing task.

42. The electronic device of claim 39, wherein identifying the computing task includes launching a digital assistant session on the second processor.

43. The electronic device of claim 42, wherein the digital assistant session identifies and executes the computing task.

44. The electronic device of claim 42, wherein launching the digital assistant session includes displaying a user interface associated with the digital assistant session.

45. The electronic device of claim 44, wherein the user interface associated with the digital assistant session is displayed in a full-screen view.

46. The electronic device of claim 42, wherein launching the digital assistant session includes activating one or more audio components on the device.

47. The electronic device of claim 42, wherein the portion of the contents of the memory buffer is provided to a remote server associated with the digital assistant session.

48. The electronic device of claim 42, the one or more programs further including instructions for:

further in accordance with the determination that the user input meets the criteria;

activating a second microphone on the device, and

streaming audio detected by the second microphone to the digital assistant session.

49. The electronic device of claim 42, wherein the user input is a user input invoking the digital assistant session.

50. The electronic device of claim 39, wherein the user input is a button press.

51. The electronic device of claim 50, wherein the criteria includes a criterion that is met when the button press exceeds a predetermined threshold duration.

52. The electronic device of claim 39, wherein the user input is audio data captured by the microphone and written to the buffer.

53. The electronic device of claim 52, wherein the criteria includes a criterion that is met when the device determines that at least a second portion of the memory buffer includes audio data representing a predetermined trigger.

54. The electronic device of claim 52, wherein the criteria includes a criterion that is met when the device determines that the audio data corresponds to an authorized user.

55. The electronic device of claim 39, wherein the audio input is captured while the second processor is in a low-power mode, the one or more programs further including instructions for:

further in accordance with the determination that the user input meets the criteria, causing the second processor to exit the low-power mode.

56. The electronic device of claim 39, wherein the first processor is a processor of a first type and the second processor is a processor of a second type.

57. The electronic device of claim 39, wherein identifying the computing task further includes identifying one or more parameters corresponding to the user intent.