Natural Language Control of Secondary Device

• US 20150154976A1
• Filed: 12/02/2013
• Published: 06/04/2015
• Est. Priority Date: 12/02/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Rawles, LLC (Amazon.com, Inc.)
• Inventors: Mutagi, Rohan
• Granted Patent: US 9,698,999 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 12/281 indicating a format for calling an appliance service function in a home automation network for remote control or remote monitoring of applications H04L67/025
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04L 12/2825 Reporting to a device located outside the home and the home network access arrangements H04L12/2856; for remote control or remote monitoring of applications H04L67/025; telephonic communication systems adapted for combination with telemetering systems H04M11/002
• CPC Class Codes: H04L 12/2829 involving user profiles according to which the execution of a home appliance functionality is automatically triggered

Abstract

Natural language controlled devices may be configured to activate command recognition in response to one or more wake words. Techniques are provided to enable a voice controlled system to detect or receive an indication of a secondary device available to be controlled. The voice controlled system communicates with the secondary device to obtain information related to the secondary device. The voice controlled system may output of an audio query requesting audio input data related to controlling the secondary device from a user and generate, based on the requested audio input data, recognition data utilized to recognize of at least part of one or more commands to issue one or more controls to the secondary device.

Claims

1. A computing system comprising:

one or more processors;

one or more non-transitory computer-readable storage media storing computer-executable instructions;

secondary device control instructions maintained in the one or more non-transitory computer-readable storage media and executed by the one or more processors to;

detect a secondary device;

communicate with the secondary device to enable the secondary device to connect to a network and to obtain information related to the secondary device;

cause output of an audio query requesting information from a user related to controlling the secondary device;

receive first audio input data comprising the requested information from the user;

cause first speech processing to be performed on the first audio input data, wherein the output of the first speech processing comprises recognition data;

receive second audio input data comprising a command for the secondary device;

cause second speech processing to be performed on the second audio input data, wherein the second speech processing uses the recognition data; and

based at least in part on the output of the second speech processing, communicate with the secondary device to cause the secondary device to execute at least part of the command.

2. The system as recited in claim 1, wherein the instruction to communicate with the secondary device cause communication via a direct connection and enable the secondary device to connect wirelessly to a local area network (LAN).

3. The system as recited in claim 1, wherein the requesting of the information from the user comprises requesting the user input at least part of a spoken identifier to be used when controlling the secondary device.

4. The system as recited in claim 1, further comprising instructions to cause output of a request to the user to test the generated recognition data by prompting the user to speak a phrase at least partially based on the first audio input data.

5. The system as recited in claim 1, further comprising instructions to:

receive third audio input data from a user comprising information relating to creating a custom command;

cause third speech processing to be performed on the third audio input data;

receive fourth audio input data from a user comprising at least one action to be performed corresponding to the custom command;

cause fourth speech processing to be performed on the fourth audio input data;

configure the custom command to invoke the first action and a second action.

6. A method comprising:

receiving an indication of a secondary device available to be controlled;

communicating with the secondary device to obtain information related to the secondary device;

causing output of an audio query requesting information from a user related to controlling the secondary device;

receiving audio input data comprising the requested information from the user; and

causing first speech processing to be performed on the audio input data, wherein the output of the first speech processing comprises recognition data.

7. The method as recited in claim 6, further comprising communicating with the secondary device to enable the secondary device to connect to a network.

8. The method as recited in claim 6, wherein the requested information comprises at least part of a spoken identifier for the secondary device.

9. The method as recited in claim 6, further comprising causing output of an additional audio query prompting the user to speak a phrase at least partially based on the audio input data to test the recognition data.

10. The method as recited in claim 6, wherein the audio query that requests information from the user related to controlling the secondary device includes information about the secondary device.

11. The method as recited in claim 10, wherein the information about the secondary device indicates a type of the secondary device.

12. The method as recited in claim 11, further comprising:

receiving second audio input data comprising a command for the secondary device;

causing second speech processing to be performed on the second audio input data, wherein the second speech processing uses the recognition data; and

based at least in part on the output of the second speech processing, communicating with the secondary device to cause the secondary device to execute at least part of the command.

13. The method as recited in claim 6, further comprising:

receiving third audio input data from a user comprising information relating to creating a custom command;

causing third speech processing to be performed on the third audio input data, the second recognized command relating to configuring a particular command to invoke functions of a plurality of secondary devices;

receiving fourth audio input data from a user comprising at least one action to be performed corresponding to the custom command;

causing fourth speech processing to be performed on the fourth audio input data; and

configuring the custom command to invoke the first action and a second action.

14. The method as recited in claim 6, wherein the secondary device is controlled using at least one protocol, the at least one protocol being at least one of Zigbee, Z-wave, Insteon, Bluetooth, Wi-Fi, or Wi-Fi Direct.

15. One or more non-transitory computer-readable storage media storing computer-executable instructions executable in at least one computing device, whereupon execution of the computer-executable instructions causes performing of operations comprising:

communicating with a secondary device to obtain information related to controlling the secondary device;

causing output of an audible request for information from a user that may be used subsequently to control the secondary device;

receiving natural language input data comprising the requested information from the user; and

cause first speech processing to be performed on the natural language input data, wherein the output of the first speech processing comprises recognition data.

16. The one or more computer-readable media as recited in claim 15, wherein the recognition data is first recognition data associated with a first user of a plurality of users of a device associated with the one or more processors;

the instructions further comprising;

instructions for storing the first recognition data such that the first recognition data is associated with the secondary device and the first user;

instructions for storing second recognition data such that the second recognition data is associated with the secondary device and a second user; and

instructions for storing third recognition data such that the third recognition data is associated with the secondary device, the first user and the second user.

17. The one or more computer-readable media as recited in claim 16, wherein the first recognition data, second recognition data and third recognition data are stored in a hash table.

18. The one or more computer-readable media as recited in claim 15, wherein the audible request for information from the user related to controlling the secondary device includes information about the secondary device.

19. The one or more computer-readable media as recited in claim 18, wherein the information about the secondary device indicates a type of the secondary device.

20. The one or more computer-readable media as recited in claim 15, the instructions further comprising instructions for communicating with the secondary device to enable the secondary device to connect to a network.

21. The one or more computer-readable media as recited in claim 15, the instructions further comprising instructions for:

receiving another natural language input data comprising a command for the secondary device;

causing second speech processing to be performed on the another natural language input data, wherein the second speech processing uses the recognition data;

based at least in part on the output of the second speech processing, communicating with the secondary device to issue at least part of the command to the secondary device;

receiving a response to the communicating with the secondary device to issue at least part of the command to the secondary device; and

causing output of an audible message to the user based at least in part on the response.