Natural language control of secondary device

• US 9,698,999 B2
• Filed: 12/02/2013
• Issued: 07/04/2017
• Est. Priority Date: 12/02/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Mutagi, Rohan
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US14/094,270
• Publication Number:
  US 20150154976A1
• Time in Patent Office: 1,310 Days
• Field of Search: 704275, 704274, 704260, 704251, 704235, 704208, 715771, 709224, 709223, 709220, 707670
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 12/281 indicating a format for calling an appliance service function in a home automation network for remote control or remote monitoring of applications H04L67/025
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04L 12/2825 Reporting to a device located outside the home and the home network access arrangements H04L12/2856; for remote control or remote monitoring of applications H04L67/025; telephonic communication systems adapted for combination with telemetering systems H04M11/002
• CPC Class Codes: H04L 12/2829 involving user profiles according to which the execution of a home appliance functionality is automatically triggered

Abstract

Natural language controlled devices may be configured to activate command recognition in response to one or more wake words. Techniques are provided to enable a voice controlled system to detect or receive an indication of a secondary device available to be controlled. The voice controlled system communicates with the secondary device to obtain information related to the secondary device. The voice controlled system may output of an audio query requesting audio input data related to controlling the secondary device from a user and generate, based on the requested audio input data, recognition data utilized to recognize of at least part of one or more commands to issue one or more controls to the secondary device.

Claims

1. A computing system comprising:

one or more processors;

one or more non-transitory computer-readable storage media storing computer-executable instructions; and

secondary device control instructions maintained in the one or more non-transitory computer-readable storage media and executed by the one or more processors to;

detect a secondary device;

communicate with the secondary device to enable the secondary device to connect to a network and to obtain information related to the secondary device;

cause, based at least in part on detection of the secondary device, output of first audio, the first audio indicating the detection of the secondary device and requesting a user to supply a name for the secondary device;

receive first audio input data comprising the name from the user;

cause first speech processing to be performed on the first audio input data, wherein an output of the first speech processing comprises recognition data;

assign the name to the secondary device;

confirm the name assigned to the secondary device by causing output of second audio, the second audio requesting the user to speak the name of the secondary device and a command for the secondary device to execute;

receive second audio input data comprising the name of the secondary device and the command for the secondary device;

cause second speech processing to be performed on the second audio input data, wherein the second speech processing uses the recognition data; and

cause, based at least in part on the second speech processing, the secondary device to execute at least part of the command.

2. The system as recited in claim 1, wherein the instruction to communicate with the secondary device includes communication via a direct connection and enables the secondary device to connect wirelessly to a local area network (LAN).

3. The system as recited in claim 1, wherein causing output of the first audio requesting the user to supply the name for the secondary device comprises requesting the user input at least part of a spoken identifier to be used when controlling the secondary device, and wherein the information identifies a type of the secondary device.

4. The system as recited in claim 1, further comprising instructions to:

receive third audio input data from the user comprising information relating to creating a custom command;

cause third speech processing to be performed on the third audio input data;

receive fourth audio input data from the user comprising at least one action to be performed corresponding to the custom command;

cause fourth speech processing to be performed on the fourth audio input data; and

configure the custom command to invoke the at least one action to be performed.

5. The system as recited in claim 1, further comprising instructions to receive a confirmation message that the secondary device is able to be controlled based at least in part on a determination that the secondary device has used the recognition data and has executed the at least the part of the command.

6. The system as recited in claim 3, wherein the information further identifies a list of functions that the secondary device is capable of executing.

7. The system as recited in claim 1, further comprising instructions to compare the command with a list of functions that the device is able to execute.

8. A method comprising:

receiving an indication of a secondary device available to be controlled;

communicating with the secondary device to obtain information related to the secondary device;

based at least in part on receiving the indication, causing output of first audio requesting a user to supply a name for the secondary device;

receiving first audio input data comprising the name from the user;

causing first speech processing to be performed on the first audio input data, wherein an output of the first speech processing comprises recognition data associated with the secondary device;

causing output of second audio, the second audio requesting that the user provide a name for the secondary device and a command for the secondary device to execute;

receiving second audio input data comprising at least the name of the secondary device and the command for the secondary device to execute;

causing second speech processing to be performed on the second audio input data, wherein the second speech processing uses the recognition data; and

confirming that the secondary device is capable of being controlled based at least in part on the secondary device executing at least part of the command.

9. The method as recited in claim 8, further comprising communicating with the secondary device to enable the secondary device to connect to a network.

10. The method as recited in claim 8, wherein the name comprises at least part of a spoken identifier for the secondary device and wherein the output of the first audio includes information indicating one or more of detection of the secondary device or a type of the secondary device.

11. The method as recited in claim 8, wherein the output of the first audio includes information about the secondary device.

12. The method as recited in claim 11, wherein the information about the secondary device indicates a type of the secondary device.

13. The method as recited in claim 8, further comprising:

receiving third audio input data from the user comprising information relating to creating a custom command;

causing third speech processing to be performed on the third audio input data, the custom command relating to configuring a particular command to invoke functions of a plurality of secondary devices;

receiving fourth audio input data from a user comprising at least one action to be performed corresponding to the custom command;

causing fourth speech processing to be performed on the fourth audio input data; and

configuring the custom command to invoke the at least one action to be performed.

14. The method as recited in claim 8, wherein the secondary device is controlled using at least one protocol, the at least one protocol being at least one of Zigbee, Z-wave, Insteon, Bluetooth, Wi-Fi, or Wi-Fi Direct.

15. One or more non-transitory computer-readable storage media storing computer-executable instructions executable in at least one computing device, whereupon execution of the computer-executable instructions causes performing of operations comprising:

detecting that a secondary device is available to be controlled;

communicating with the secondary device to obtain information related to controlling the secondary device;

based at least in part on detecting that the secondary device is available to be controlled, causing output of a first request for a user to supply a name that is to be used subsequently to control the secondary device;

receiving first natural language input data comprising the name from the user;

cause first speech processing to be performed on the first natural language input data, wherein an output of the first speech processing comprises recognition data;

assigning the name to the secondary device; and

confirming the name assigned to the secondary device, the confirming based at least in part on;

causing output of a second request, the second request requesting the user to provide the name of the secondary device and a command for the secondary device to execute,

receiving second natural language input data comprising at least the name of the secondary device and the command for the secondary device to execute,

causing second speech processing to be performed on the second natural language input data, the second speech processing using the recognition data, and

causing the secondary device to execute at least part of the command.

16. The one or more non-transitory computer-readable media as recited in claim 15, wherein the user is a first user, and the recognition data is first recognition data associated with the first user of a plurality of users of a device associated with the one or more processors, and wherein the instructions further comprising instructions for:

storing the first recognition data such that the first recognition data is associated with the secondary device and the first user;

storing second recognition data such that the second recognition data is associated with the secondary device and a second user of the plurality of users; and

storing third recognition data such that the third recognition data is associated with the secondary device, the first user, and the second user.

17. The one or more non-transitory computer-readable media as recited in claim 16, wherein the first recognition data, the second recognition data, and the third recognition data are stored in a hash table.

18. The one or more non-transitory computer-readable media as recited in claim 15, wherein the output of the first request includes information about the secondary device and a type of the secondary device.

19. The one or more non-transitory computer-readable media as recited in claim 15, the instructions further comprising instructions for communicating with the secondary device to enable the secondary device to connect to a network.

20. The one or more non-transitory computer-readable media as recited in claim 15, wherein the command is a first command and the instructions further comprising instructions for:

receiving third natural language input data comprising a second command for the secondary device;

causing third speech processing to be performed on the third natural language input data, wherein the third speech processing uses the recognition data;

based at least in part on an output of the third speech processing, communicating with the secondary device to issue at least part of the second command to the secondary device;

receiving a response to issue at least part of the second command to the secondary device; and

causing output of an audible message to the user based at least in part on the response.

21. The one or more non-transitory computer-readable media as recited in claim 15, wherein confirming the name assigned to the secondary device further includes causing output of a confirmation message that indicates that the secondary device is available to be controlled.