ZERO LATENCY DIGITAL ASSISTANT

• US 20170068513A1
• Filed: 05/05/2016
• Published: 03/09/2017
• Est. Priority Date: 09/08/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: STASIOR, William F., CARSON, David A., DASARI, Rohit, KIM, Yoon
• Granted Patent: US 10,747,498 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/0673 Single storage device
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/285 Memory allocation or algorithm optimisation to reduce hardware requirements
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04M 2201/40 using speech recognition
• CPC Class Codes: H04M 2250/74 with voice recognition means

Abstract

An electronic device can implement a zero-latency digital assistant by capturing audio input from a microphone and using a first processor to write audio data representing the captured audio input to a memory buffer. In response to detecting a user input while capturing the audio input, the device can determine whether the user input meets a predetermined criteria. If the user input meets the criteria, the device can use a second processor to identify and execute a task based on at least a portion of the contents of the memory buffer.

Claims

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a first processor and a second processor of an electronic device with a microphone cause the device to:

capture audio input from the microphone;

write, using the first processor, data representing the captured audio input to a memory buffer;

detect a user input while capturing audio input using the microphone;

determine, using the first processor, whether the user input has met predetermined criteria;

in accordance with a determination that the user input has met the criteria, identify, using the second processor, a computing task based on at least a portion of the contents of the memory buffer; and

in accordance with a determination that the user input has not met the criteria, forgo identifying the task.

2. The non-transitory computer-readable storage medium according to claim 1, wherein the at least a portion of the contents of the memory buffer includes data written to the buffer before the user input is detected.

3. The non-transitory computer-readable storage medium according to claim 1, wherein the at least a portion of the contents of the memory buffer includes data written to the buffer during the detection of the user input.

4. The non-transitory computer-readable storage medium according to claim 1, further comprising instructions to cause the device to:

further in accordance with a determination that the user input has met the criteria, execute the identified computing task.

5. The non-transitory computer-readable storage medium according to claim 1, wherein identifying the computing task includes launching a digital assistant session on the second processor.

6. The non-transitory computer-readable storage medium according to claim 5, wherein the digital assistant identifies and executes the computing task.

7. The non-transitory computer-readable storage medium according to claim 5, wherein launching the digital assistant session includes displaying a user interface associated with the digital assistant session.

8. The non-transitory computer-readable storage medium according to claim 7, wherein the user interface associated with the digital assistant is displayed in a full-screen view.

9. The non-transitory computer-readable storage medium according to claim 5, wherein launching the digital assistant session includes activating one or more audio components on the device.

10. The non-transitory computer-readable storage medium according to claim 5, wherein the at least a portion of the contents of the memory buffer is provided to a remote server associated with the digital assistant.

11. The non-transitory computer-readable storage medium according to claim 1, further comprising instructions to cause the device to:

further in accordance with a determination that the user input has met the criteria;

activate a second microphone on the device, and

stream audio detected by the second microphone to the digital assistant.

12. The non-transitory computer-readable storage medium according to claim 1, wherein the user input is a button press.

13. The non-transitory computer-readable storage medium according to claim 12, wherein the criteria includes a criterion that is met when the button press exceeds a predetermined threshold duration.

14. The non-transitory computer-readable storage medium according to claim 1, wherein the user input is audio data captured by the microphone and written to the buffer.

15. The non-transitory computer-readable storage medium according to claim 14, wherein the criteria includes a criterion that is met when the device determines that at least a second portion of the memory buffer includes audio data representing a predetermined trigger.

16. The non-transitory computer-readable storage medium according to claim 14, wherein the criteria includes a criterion that is met when the device determines that the audio data corresponds to an authorized user.

17. The non-transitory computer-readable storage medium according to claim 1, wherein the audio input is captured while the second processor is in a low-power mode, further comprising instructions to cause the device to:

further in accordance with the determination that the user input has met the criteria, cause the second processor to exit the low-power mode.

18. A method, comprising:

at an electronic device including a microphone, a first processor, and a second processor;

capturing audio input from the microphone;

writing, using the first processor, data representing the captured audio input to a memory buffer;

detecting a user input while capturing audio input using the microphone;

determining, using the first processor, whether the user input has met predetermined criteria;

in accordance with a determination that the user input has met the criteria, identifying, using the second processor, a computing task based on at least a portion of the contents of the memory buffer; and

in accordance with a determination that the user input has not met the criteria, forgoing identifying the task.

19. An electronic device, comprising:

a microphone;

two or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the two or more processors, the one or more programs including instructions for;

capturing audio input from the microphone;

writing, using a first processor, data representing the captured audio input to a memory buffer in the memory;

detecting a user input while capturing audio input using the microphone;

determining, using the first processor, whether the user input has met predetermined criteria;

in accordance with a determination that the user input has met the criteria, identifying, using a second processor, a computing task based on at least a portion of the contents of the memory buffer; and

in accordance with a determination that the user input has not met the criteria, forgoing identifying the task.