Local persisting of data for selectively offline capable voice action in a voice-enabled electronic device

• US 10,083,697 B2
• Filed: 05/27/2015
• Issued: 09/25/2018
• Est. Priority Date: 05/27/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sung, Sangsoo, Gao, Yuli, Murugesan, Prathab
• Primary Examiner(s): Thomas-Homescu, Anne L
• Application Number:
  US14/723,279
• Publication Number:
  US 20160351200A1
• Time in Patent Office: 1,217 Days
• CPC Class Codes: G06F 16/3343 using phonetics
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: H04L 67/01 Protocols

Abstract

Data associated with a selectively offline capable voice action is locally persisted in a voice-enabled electronic device whenever such an action cannot be competed locally due to the device being offline to enable the action to later be completed after online connectivity has been restored. Synchronization with an online service and/or another electronic device, and/or retrieval of context sensitive data from an online service may be performed after online connectivity has been restored to enable the voice action to thereafter be completed.

Claims

1. A method, comprising:

receiving a voice input with a voice-enabled electronic device; and

in the voice-enabled electronic device, and responsive to receiving at least a portion of the voice input;

performing local processing of the at least a portion of the voice input, wherein the local processing identifies that the voice input is associated with a selectively offline capable voice action and builds at least a portion of the selectively offline capable voice action, and wherein the local processing identifies that the voice input is associated with the selectively offline capable voice action by identifying that the voice input is associated with a voice action that is of a type that in at least one instance is capable of being completed offline and in at least one other instance can only be completed using a connection with an online service because a parameter for the voice action is unresolvable strictly based upon data provided in the voice input and/or data available locally on the voice-enabled device;

determining during the local processing and when the voice-enabled electronic device is offline whether the selectively offline capable voice action can be completed offline, wherein determining whether the selectively offline capable voice action can be completed offline includes determining during the local processing whether the selectively offline capable voice action includes at least one parameter that remains unresolved from the voice input;

in response to determining that the selectively offline capable voice action can be completed offline, completing the selectively offline capable voice action with the local processing; and

in response to determining that the selectively offline capable voice action cannot be completed offline;

locally persisting data generated by the local processing for the selectively offline capable voice action; and

completing the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service, including retrieving a value for the at least one parameter from the online service and, after retrieving the value from the online service, locally completing the selectively offline capable voice action using the locally persisted data and the retrieved value.

2. The method of claim 1, wherein the voice-enabled electronic device comprises a mobile device configured to communicate with the online service when in communication with a wireless network.

3. The method of claim 1, wherein performing the local processing dynamically builds the at least a portion of the offline capable voice action prior to completely receiving the voice input with the voice-enabled electronic device, and wherein determining during the local processing and when the voice-enabled electronic device is offline whether the selectively offline capable voice action can be completed offline is performed prior to completely receiving the voice input with the voice-enabled electronic device.

4. The method of claim 1, wherein completing the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service includes retrieving context sensitive data from the online service and completing the selectively offline capable voice action using the context sensitive data.

5. The method of claim 4, wherein completing the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service includes issuing a query to the online service for the context sensitive data.

6. The method of claim 5, wherein the offline capable voice action includes at least one location sensitive parameter, wherein the query includes a current location of the voice-enabled electronic device, and wherein the context sensitive data includes location sensitive data.

7. The method of claim 1, wherein completing the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service includes synchronizing the voice-enabled electronic device with the online service after performing the local completion.

8. The method of claim 1, wherein completing the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service includes synchronizing the voice-enabled electronic device with at least one other electronic device associated with the user of the voice-enabled electronic device after performing the local completion.

9. The method of claim 1, wherein locally persisting data generated by the local processing for the selectively offline capable voice action includes locally persisting diagnostic data associated with the selectively offline capable voice action, and wherein completing the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service includes uploading the diagnostic data associated with the selectively offline capable voice action.

10. The method of claim 1, further comprising, in response to determining that the selectively offline capable voice action cannot be completed offline, performing a local completion of the selectively offline capable voice action on the voice-enabled electronic device, wherein completing the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service includes performing a remote completion of the selectively-offline capable voice action with the online service.

11. The method of claim 1, further comprising, in response to determining that the selectively offline capable voice action cannot be completed offline, notifying a user of the voice-enabled electronic device that the selectively offline capable voice action will be completed after the voice-enabled electronic device is connected with the online service.

12. The method of claim 11, further comprising, in response to determining that the selectively offline capable voice action cannot be completed offline, notifying the user of the voice-enabled electronic device that the selectively offline capable voice action has been completed after the voice-enabled electronic device has connected with the online service.

13. The method of claim 1, wherein the type of the selectively offline capable voice action is selected from the group consisting of a reminder type, a communication type, an event type, a device setting type, a media control type and a navigation type.

14. An apparatus including memory and one or more processors operable to execute instructions stored in the memory, comprising instructions to:

receive a voice input with a voice-enabled electronic device; and

in the voice-enabled electronic device, and responsive to receiving at least a portion of the voice input;

perform local processing of the at least a portion of the voice input, wherein the local processing identifies that the voice input is associated with a selectively offline capable voice action and builds at least a portion of the selectively offline capable voice action, and wherein the local processing identifies that the voice input is associated with the selectively offline capable voice action by identifying that the voice input is associated with a voice action that is of a type that in at least one instance is capable of being completed offline and in at least one other instance can only be completed using a connection with an online service because a parameter for the voice action is unresolvable strictly based upon data provided in the voice input and/or data available locally on the voice-enabled device;

determine during the local processing and when the voice-enabled electronic device is offline whether the selectively offline capable voice action can be completed offline, wherein the instructions to determine whether the selectively offline capable voice action can be completed offline includes instructions to determine during the local processing whether the selectively offline capable voice action includes at least one parameter that remains unresolved from the voice input;

in response to a determination that the selectively offline capable voice action can be completed offline, complete the selectively offline capable voice action with the local processing; and

in response to determining that the selectively offline capable voice action cannot be completed offline;

locally persist data generated by the local processing for the selectively offline capable voice action; and

complete the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service, including instructions to retrieve a value for the at least one parameter from the online service and, after retrieving the value from the online service, locally complete the selectively offline capable voice action using the locally persisted data and the retrieved value.

15. The apparatus of claim 14, wherein the instructions include:

first instructions implementing a streaming voice to text module that converts a digital audio signal of the voice input to text, wherein the first instructions dynamically generate a plurality of text tokens from the digital audio signal; and

second instructions implementing a streaming semantic processor that dynamically builds the portion of the selectively offline capable voice action from at least a portion of the plurality of text tokens.

16. The apparatus of claim 14, wherein the instructions that complete the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service include instructions that retrieve context sensitive data from the online service and complete the selectively offline capable voice action using the context sensitive data.

17. The apparatus of claim 14, wherein the instructions that complete the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service include instructions that synchronize the voice-enabled electronic device with an online service or with at least one other electronic device associated with the user of the voice-enabled electronic device.

18. The apparatus of claim 14, further comprising instructions that, in response to determining that the selectively offline capable voice action cannot be completed offline, perform a local completion of the selectively offline capable voice action on the voice-enabled electronic device, and wherein the instructions that complete the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service include instructions that perform a remote completion of the selectively-offline capable voice action with the online service.

19. A non-transitory computer readable storage medium storing computer instructions executable by one or more processors to perform a method comprising:

receiving a voice input with a voice-enabled electronic device; and

in the voice-enabled electronic device, and responsive to receiving at least a portion of the voice input;

performing local processing of the at least a portion of the voice input, wherein the local processing identifies that the voice input is associated with a selectively offline capable voice action and builds at least a portion of the selectively offline capable voice action, and wherein the local processing identifies that the voice input is associated with the selectively offline capable voice action by identifying that the voice input is associated with a voice action that is of a type that in at least one instance is capable of being completed offline and in at least one other instance can only be completed using a connection with an online service because a parameter for the voice action is unresolvable strictly based upon data provided in the voice input and/or data available locally on the voice-enabled device;

determining during the local processing and when the voice-enabled electronic device is offline whether the selectively offline capable voice action can be completed offline, wherein determining whether the selectively offline capable voice action can be completed offline includes determining during the local processing whether the selectively offline capable voice action includes at least one parameter that remains unresolved from the voice input;

in response to determining that the selectively offline capable voice action can be completed offline, completing the selectively offline capable voice action with the local processing; and

in response to determining that the selectively offline capable voice action cannot be completed offline;

locally persisting data generated by the local processing for the selectively offline capable voice action; and

completing the selectively offline capable voice action using the locally persisted data after the voice-enabled electronic device is connected with the online service, including retrieving a value for the at least one parameter from the online service and, after retrieving the value from the online service, locally completing the selectively offline capable voice action using the locally persisted data and the retrieved value.