Voice interaction at a primary device to access call functionality of a companion device

• US 10,720,160 B2
• Filed: 07/08/2019
• Issued: 07/21/2020
• Est. Priority Date: 06/01/2018
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Schramm, Karl Ferdinand, Binder, Justin, Phipps, Benjamin S., Sung, Po Keng
• Primary Examiner(s): Pham, Thierry L
• Application Number:
  US16/504,782
• Publication Number:
  US 20190371331A1
• Time in Patent Office: 379 Days
• Field of Search: 704257
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04M 2201/40 using speech recognition speech recognition per se G10L15/00
• CPC Class Codes: H04M 3/42212 Call pickup comprising simultaneous alerting H04M3/46
• CPC Class Codes: H04M 3/42263 where the same subscriber uses different terminals, i.e. nomadism
• CPC Class Codes: H04M 3/527 Centralised call answering arrangements not requiring operator intervention

Abstract

The present disclosure generally relates to using voice interaction to access call functionality of a companion device. In an example process, a user utterance is received. Based on the user utterance and contextual information, the process causes a server to determine a user intent corresponding to the user utterance. The contextual information is based on a signal received from the companion device. In accordance with the user intent corresponding to an actionable intent of answering the incoming call, a command is received. Based on the command, instructions are provided to the companion device, which cause the companion device to answer the incoming call and provide audio data of the answered incoming call. Audio is outputted according to the audio data of the answered incoming call.

Claims

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device with a speaker, cause the electronic device to:

receive a user utterance representing a request for call-related information stored on a companion device;

cause a server to determine, based on the user utterance, a user intent of providing call-related information;

receive, from the server, a command representing a task to satisfy the user intent of providing call-related information;

determine, based on the command and contextual information, that the companion device is to execute the command;

in accordance with a determination that the companion device is to execute the command, provide instructions to the companion device to retrieve the call-related information and send the call-related information to the electronic device; and

in accordance with the instructions successfully causing the companion device to retrieve and send the call-related information to the electronic device, provide an audio output at the speaker, wherein the audio output includes the call-related information received from the companion device.

2. The non-transitory computer-readable storage medium of claim 1, wherein the call-related information comprises information stored on the companion device based on one or more calls received by the companion device prior to the electronic device receiving the user utterance.

3. The non-transitory computer-readable storage medium of claim 1, wherein the call-related information comprises a voicemail audio file stored on the companion device, and

wherein the voicemail audio file comprises call information for a voicemail and an audio recording of the voicemail.

4. The non-transitory computer-readable storage medium of claim 3, wherein the audio output at the speaker comprises a first spoken dialogue and the audio recording of the voicemail, and

wherein the first spoken dialogue comprises the call information for the voicemail.

5. The non-transitory computer-readable storage medium of claim 4, wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

after receiving the call-related information at the electronic device and prior to providing the audio output at the speaker;

send, to the server, a representation of the call information for the voicemail; and

receive, from the server, the first spoken dialogue comprising the call information.

6. The non-transitory computer-readable storage medium of claim 1, wherein the call-related information comprises caller information stored on the companion device for one or more missed calls.

7. The non-transitory computer-readable storage medium of claim 6, wherein the audio output at the speaker comprises a second spoken dialogue, and

wherein the second spoken dialogue comprises the caller information corresponding to the one or more missed calls.

8. The non-transitory computer-readable storage medium of claim 1, wherein the command specifies an actionable intent node corresponding to the user intent, and wherein the instructions include the command.

9. The non-transitory computer-readable storage medium of claim 1, wherein the contextual information indicates that the companion device has telephone capabilities.

10. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

prior to receiving the user utterance, exchange authentication information with the companion device; and

in accordance with a determination, based on the authentication information, that the companion device is a registered device of the electronic device, establish a wireless communication connection with the companion device,

wherein the instructions are provided via the established wireless communication connection, and

wherein the call-related information is received via the established wireless communication connection.

11. The non-transitory computer-readable storage medium of claim 10, wherein the contextual information indicates that the wireless communication connection is established between the electronic device and the companion device and that the companion device is a registered device of the electronic device.

12. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

after providing the instructions to the companion device, receive a confirmation signal from the companion device, wherein the confirmation signal indicates whether the call-related information was successfully retrieved; and

provide a representation of the confirmation signal to the server.

13. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs comprise further instructions, which when executed by the one or more processors of the electronic device, cause the electronic device to:

determine whether a first portion of the user utterance contains a predefined spoken trigger; and

in accordance with a determination that the first portion of the user utterance contains a predefined spoken trigger, receive a second portion of the user utterance separate from the first portion, wherein the user intent is determined from the second portion of the user utterance.

14. An electronic device, comprising:

a speaker;

one or more processors; and

memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving a user utterance representing a request for call-related information stored on a companion device;

causing a server to determine, based on the user utterance, a user intent of providing call-related information;

receiving, from the server, a command representing a task to satisfy the user intent of providing call-related information;

determining, based on the command and contextual information, that the companion device is to execute the command;

in accordance with a determination that the companion device is to execute the command, providing instructions to the companion device to retrieve the call-related information and send the call-related information to the electronic device; and

in accordance with the instructions successfully causing the companion device to retrieve and send the call-related information to the electronic device, providing an audio output at the speaker, wherein the audio output includes the call-related information received from the companion device.

16. The electronic device of claim 14, wherein the call-related information comprises information stored on the companion device based on one or more calls received by the companion device prior to the electronic device receiving the user utterance.

17. The electronic device of claim 14, wherein the call-related information comprises a voicemail audio file stored on the companion device, and

wherein the voicemail audio file comprises call information for a voicemail and an audio recording of the voicemail.

18. The electronic device of claim 17, wherein the audio output at the speaker comprises a first spoken dialogue and the audio recording of the voicemail, and

wherein the first spoken dialogue comprises the call information for the voicemail.

19. The electronic device of claim 18, wherein the one or more programs further include instructions for:

after receiving the call-related information at the electronic device and prior to providing the audio output at the speaker;

sending, to the server, a representation of the call information for the voicemail; and

receiving, from the server, the first spoken dialogue comprising the call information.

20. The electronic device of claim 14, wherein the call-related information comprises caller information stored on the companion device for one or more missed calls.

21. The electronic device of claim 20, wherein the audio output at the speaker comprises a second spoken dialogue, and

wherein the second spoken dialogue comprises the caller information corresponding to the one or more missed calls.

22. The electronic device of claim 14, wherein the command specifies an actionable intent node corresponding to the user intent, and wherein the instructions include the command.

23. The electronic device of claim 14, wherein the contextual information indicates that the companion device has telephone capabilities.

24. The electronic device of claim 14, wherein the one or more programs further include instructions for:

prior to receiving the user utterance, exchanging authentication information with the companion device; and

in accordance with a determination, based on the authentication information, that the companion device is a registered device of the electronic device, establishing a wireless communication connection with the companion device,

wherein the instructions are provided via the established wireless communication connection, and

wherein the call-related information is received via the established wireless communication connection.

25. The electronic device of claim 24, wherein the contextual information indicates that the wireless communication connection is established between the electronic device and the companion device and that the companion device is a registered device of the electronic device.

26. The electronic device of claim 14, wherein the one or more programs further include instructions for:

after providing the instructions to the companion device, receiving a confirmation signal from the companion device, wherein the confirmation signal indicates whether the call-related information was successfully retrieved; and

providing a representation of the confirmation signal to the server.

27. The electronic device of claim 14, wherein the one or more programs further include instructions for:

determining whether a first portion of the user utterance contains a predefined spoken trigger; and

in accordance with a determination that the first portion of the user utterance contains a predefined spoken trigger, receiving a second portion of the user utterance separate from the first portion, wherein the user intent is determined from the second portion of the user utterance.

15. A method for using voice interaction to access call-related information of a companion device, the method comprising:

at an electronic device having a processor, memory, and a speaker;

receiving a user utterance representing a request for call-related information stored on the companion device;

causing a server to determine, based on the user utterance, a user intent of providing call-related information;

receiving, from the server, a command representing a task to satisfy the user intent of providing call-related information;

determining, based on the command and contextual information, that the companion device is to execute the command;

in accordance with a determination that the companion device is to execute the command, providing instructions to the companion device to retrieve the call-related information and send the call-related information to the electronic device; and

in accordance with the instructions successfully causing the companion device to retrieve and send the call-related information to the electronic device, providing an audio output at the speaker, wherein the audio output includes the call-related information received from the companion device.

28. The method of claim 15, wherein the call-related information comprises information stored on the companion device based on one or more calls received by the companion device prior to the electronic device receiving the user utterance.

29. The method of claim 15, wherein the call-related information comprises a voicemail audio file stored on the companion device, and

wherein the voicemail audio file comprises call information for a voicemail and an audio recording of the voicemail.

30. The method of claim 29, wherein the audio output at the speaker comprises a first spoken dialogue and the audio recording of the voicemail, and

wherein the first spoken dialogue comprises the call information for the voicemail.

31. The method of claim 30, further comprising:

after receiving the call-related information at the electronic device and prior to providing the audio output at the speaker;

sending, to the server, a representation of the call information for the voicemail; and

receiving, from the server, the first spoken dialogue comprising the call information.

32. The method of claim 15, wherein the call-related information comprises caller information stored on the companion device for one or more missed calls.

33. The method of claim 32, wherein the audio output at the speaker comprises a second spoken dialogue, and

wherein the second spoken dialogue comprises the caller information corresponding to the one or more missed calls.

34. The method of claim 15, wherein the command specifies an actionable intent node corresponding to the user intent, and wherein the instructions include the command.

35. The method of claim 15, wherein the contextual information indicates that the companion device has telephone capabilities.

36. The method of claim 15, further comprising:

prior to receiving the user utterance, exchanging authentication information with the companion device; and

in accordance with a determination, based on the authentication information, that the companion device is a registered device of the electronic device, establishing a wireless communication connection with the companion device,

wherein the instructions are provided via the established wireless communication connection, and

wherein the call-related information is received via the established wireless communication connection.

37. The method of claim 36, wherein the contextual information indicates that the wireless communication connection is established between the electronic device and the companion device and that the companion device is a registered device of the electronic device.

38. The method of claim 15, further comprising:

after providing the instructions to the companion device, receiving a confirmation signal from the companion device, wherein the confirmation signal indicates whether the call-related information was successfully retrieved; and

providing a representation of the confirmation signal to the server.

39. The method of claim 15, further comprising:

determining whether a first portion of the user utterance contains a predefined spoken trigger; and

in accordance with a determination that the first portion of the user utterance contains a predefined spoken trigger, receiving a second portion of the user utterance separate from the first portion, wherein the user intent is determined from the second portion of the user utterance.