Architecture for a hub configured to control a second device while a connection to a remote system is unavailable

• US 10,713,007 B2
• Filed: 12/12/2017
• Issued: 07/14/2020
• Est. Priority Date: 12/12/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Aiken, Mark Aran, Pasko, Stanislaw Ignacy, Soyannwo, Olusanya Temitope, Salgaonkar, Vibhav Hemant, Fineberg, Adam Barry, Webster, Roger Robert, Damle, Makarand, Mutagi, Rohan, Lee, Philip Alexander
• Primary Examiner(s): Tzeng, Feng-Tzer
• Application Number:
  US15/839,750
• Publication Number:
  US 20190179610A1
• Time in Patent Office: 945 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04L 12/2829 involving user profiles according to which the execution of a home appliance functionality is automatically triggered
• CPC Class Codes: H04L 12/2834 Switching of information between an external network and a home network access arrangements H04L12/2856
• CPC Class Codes: H04M 1/72415 for remote control of appliances
• CPC Class Codes: H04M 2250/74 with voice recognition means voice activated dialling H04M1/271; voice recognition algorithms G10L15/00

Abstract

A hub is configured to provide voice control without assistance from a remote system, which allows the hub to provide a user with the ability to control second devices in an environment by issuing voice commands, even when the hub is unable to communicate with the remote system over a wide area network (e.g., the Internet). The hub is also configured to execute rules without assistance from the remote system, which allows the hub to execute rules, even when the hub is unable to communicate with the remote system over a wide area network (e.g., the Internet).

Claims

1. A first device comprising:

one or more processors;

memory;

computer-executable instructions stored in the memory and executable by the one or more processors to cause the first device to;

determine that audio data representing user speech is to be processed by a local speech processing component that is executing on the first device;

send the audio data to the local speech processing component, the audio data having been generated by the first device;

determine intent data by processing the audio data using automatic speech recognition (ASR) and natural language understanding (NLU);

determine that the intent data is associated with a domain;

determine text data representing a name of a second device by processing the audio data using the ASR and the NLU;

lookup, in a device store maintained in the memory, an identifier corresponding to the name of the second device, the identifier corresponding to the second device that is collocated in an environment with the first device;

generate, based at least in part on the intent data and the domain, directive data that includes the identifier and an operation to be performed by the second device;

translate the directive data into a command that instructs the second device to perform the operation;

send, via a wireless protocol to the second device, or a different device in the environment, the command instructing the second device to perform the operation;

determine that a rule stored in the memory is triggered based at least in part on performance of the operation at the second device; and

cause performance of an additional operation specified in the rule in response to determining that the rule is triggered.

2. The first device of claim 1, wherein the local speech processing component is executable to generate the directive data in a format that includes a header object in JavaScript-based syntax, an endpoint object in the JavaScript-based syntax, and a payload object in the JavaScript-based syntax.

3. The first device of claim 1, wherein the computer-executable instructions are further executable by the one or more processors to cause the first device to:

send, over a wide area network, the audio data to a remote speech processing component of a remote system in addition to sending the audio data to the local speech processing component,

determine that the remote system is unavailable to the first device based at least in part on a lapse of a predefined period of time without receiving a response from the remote system within the predefined period of time.

4. A method, comprising:

receiving, at a first device, audio data that represents user speech;

determining, by the first device, that a speech processing component executing on the first device is to process the audio data;

inputting, by the first device based at least in part on the determining, the audio data to the speech processing component;

determining, by the speech processing component and based at least in part on processing the audio data, intent data;

determining a second device based at least in part on at least one of context information or the processing of the audio data;

determining an identifier, stored in memory of the first device, corresponding to the second device;

generating directive data based at least in part on the intent data, the directive data including the identifier;

sending, by the first device and to the second device, or a different device collocated in an environment with the first device, a command instructing an operation to be performed at the second device;

determining that a rule stored in the memory of the first device is triggered based at least in part on performance of the operation at the second device; and

causing performance of an additional operation specified in the rule in response to the determining that the rule is triggered.

5. The method of claim 4, wherein the generating of the directive data further comprises generating the directive data in a format used by a remote speech processing component to generate other directive data that is sent from the remote speech processing component to the first device.

6. The method of claim 4, further comprising:

sending, by the first device, the audio data to a remote speech processing component executing on a remote system in addition to the inputting of the audio data to the speech processing component.

7. The method of claim 6, wherein the directive data is first directive data, the method further comprising:

receiving, from the remote system, second directive data that includes the identifier and the operation, wherein the first directive data includes a different operation; and

discarding, based at least in part on the receiving of the second directive data, the first directive data

wherein the command sent to the second device, or the different device, is based at least in part on the second directive data and instructs the operation, to be performed at the second device.

8. The method of claim 6, further comprising:

after the sending of the audio data to the remote speech processing component, waiting a period of time for a response from the remote system; and

determining that the remote system is unavailable based at least in part on a lapse of the period of time without receiving the response at the first device within the period of time.

9. The method of claim 4, further comprising:

determining that a remote system configured to process speech is unavailable to the first device; and

determining a confidence score associated with the determining that the remote system is unavailable to the first device,

wherein the inputting the audio data to the speech processing component is further based on the confidence score satisfying a threshold confidence score.

10. The method of claim 4, further comprising, prior to the receiving of the audio data at the first device:

receiving, over a wide area network from a remote system, information about second devices that are registered to a user account to which the first device is also registered; and

storing, in the memory of the first device, the information about the second devices for identifying individual ones of the second devices to be controlled.

11. The method of claim 4, wherein the determining the intent data further comprises:

performing, by the speech processing component, automatic speech recognition (ASR) on the audio data to generate ASR text data corresponding to the audio data;

performing, by the speech processing component, natural language understanding (NLU) on the ASR text data to;

determine the intent data; and

determine that the intent data is associated with a domain.

12. The method of claim 4, further comprising, prior to the receiving of the audio data at the first device:

receiving, over a wide area network from a remote system, the rule, the rule having been assigned to the first device for execution of the rule by the first device; and

storing, in the memory of the first device, the rule for execution by the first device when the rule is triggered.

13. A first device comprising:

one or more processors; and

memory storing computer-executable instructions that, when executed by the one or more processors, cause the first device to;

determine that audio data representing user speech is to be processed by the first device;

determine, based at least in part on processing the audio data by the first device, intent data;

determine a second device based at least in part on at least one of context information or the processing of the audio data;

determine an identifier, stored in the memory, corresponding to the second device;

generate, based at least in part on the intent data, directive data that includes the identifier;

send, to the second device, or a different device collocated in an environment with the first device, a command instructing an operation to be performed at the second device;

determine that a rule stored in the memory is triggered based at least in part on performance of the operation at the second device; and

cause performance of an additional operation specified in the rule in response to determining that the rule is triggered.

14. The first device of claim 13, wherein the computer-executable instructions, when executed by the one or more processors, further cause the first device, prior to the processing of the audio data, to at least one of:

receive the audio data from a second speech interface device that is collocated in the environment with the first device;

or

generate the audio data based at least in part on an utterance captured by one or more microphones of the first device.

15. The first device of claim 13, wherein:

the command is sent to the second device; and

sending the command to the second device comprises sending the command to the second device using a wireless communications protocol that does not involve transmitting the command over a wide area network.

16. The first device of claim 13, wherein:

the different device is a third device that is within wireless communication range of the second device;

the command is sent to the different device; and

sending the command to the different device comprises sending the command to the third device using a wireless communications protocol that does not involve transmitting the command over a wide area network.

17. The first device of claim 13, wherein the computer-executable instructions, when executed by the one or more processors, further cause the first device to generate the directive data in a format used by a remote speech processing component to generate other directive data sent from the remote speech processing component to the first device.

18. The first device of claim 13, wherein the computer-executable instructions, when executed by the one or more processors, further cause the first device to:

send, over a wide area network, the audio data to a remote speech processing component of a remote system in addition to the processing of the audio data by the first device;

receive, over the wide area network and from the remote system, second directive data that includes the identifier and the operation, wherein the directive data generated by the first device includes a different operation; and

discard the directive data generated by the first device,

wherein the command sent to the second device, or the different device, is based at least in part on the second directive data and includes the operation.

19. The first device of claim 13, wherein the computer-executable instructions, when executed by the one or more processors, further cause the first device to:

send, over a wide area network, the audio data to a remote speech processing component of a remote system in addition to the processing of the audio data by the first device;

after sending the audio data to the remote speech processing component, wait a period of time for a response from the remote system; and

determine that the remote system is unavailable based at least in part on a lapse of the period of time without receiving the response at the first device within the period of time.