Remote execution of secondary-device drivers

• US 10,453,461 B1
• Filed: 06/26/2015
• Issued: 10/22/2019
• Est. Priority Date: 03/17/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Mutagi, Rohan, Whiteley, Michael Dale, Lu, He, Butler, Brian James, Torok, Fred, Lew Yuk Vong, Willy, Bronaugh, David Ross, Nies, Christopher Ryan, Sitoke, Shikher
• Primary Examiner(s): Kim, Jonathan C
• Application Number:
  US14/752,134
• Time in Patent Office: 1,579 Days
• Field of Search: None
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00

Abstract

Techniques for remotely executing a secondary-device driver for generating commands for a secondary device are described herein. For instance, a secondary device (or “appliance”) may reside within an environment, along with a device to which the secondary device communicatively couples. The device may be configured to send control signals to the secondary device for causing the secondary device to perform certain operations. For instance, a user in the environment may provide, to the device, a request that the secondary device perform a certain operation. The device, which may lack some or all of a device driver associated with the secondary device, may then work with a remote service that executes the device driver for the purpose of receiving a command from the device driver and sending the command along to the secondary device. Upon receiving the command, the secondary device may perform the operation.

Claims

1. A system comprising:

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising;

receiving, over a network and by one or more network-based devices that include the one or more processors and are remote from an environment, an audio signal generated by at least one microphone of a first device residing in the environment, the environment including the first device, a secondary device, and a third device;

receiving an identifier associated with at least one of the first device or a user account associated with the first device;

performing, at the one or more network-based devices, speech recognition on the audio signal to generate text;

identifying, from the text, a request to perform an operation on the secondary device;

identifying, based at least in part on the identifier, the user account associated with the first device;

identifying a set of secondary devices associated with the user account;

identifying, based at least in part on the request and the set of secondary devices, the secondary device to perform the operation;

identifying a device driver configured to generate commands executable by the secondary device;

sending, to the device driver, information indicative of the request to perform the operation on the secondary device, wherein the device driver is stored in the one or more network-based devices that are remote from the environment;

causing the device driver to execute to generate a command configured to cause the secondary device to perform the operation;

identifying a communication protocol utilized by the secondary device;

determining that the third device, associated with the user account and residing in the environment, supports the communication protocol;

receiving, from the device driver, the command to issue to the secondary device to cause the secondary device to perform the operation; and

sending, over the network and based at least in part on the determining, the command to one or more of the first device or the third device for sending, by the third device via the communication protocol, to the secondary device for causing the secondary device to perform the operation.

2. The system as recited in claim 1, the acts further comprising:

identifying a verbal description of the secondary device from the text; and

determining that the verbal description corresponds to the secondary device.

5. The method as recited in claim 2, wherein the receiving of the request comprises:

receiving an audio signal generated by at least one microphone of the first device;

performing speech recognition on the audio signal to generate text; and

identifying the request from the text.

6. The method as recited in claim 5, further comprising translating the request identified via the text into the information indicative of the request, the information indicative of the request being actionable by the device driver to generate the command.

7. The method as recited in claim 2, wherein the receiving of the request comprises:

receiving at least one image generated by at least one camera of the first device; and

identifying the request from the at least one image, the request comprising a gesture of a user.

8. The method as recited in claim 2, wherein the first device and the secondary device are in communication with one another over at least one wireless communication protocol.

3. A method comprising:

receiving, over a network, by one or more network-based devices, and from a first device residing within an environment remote from the one or more network-based devices, a request to perform an operation on a device in communication with the first device;

identifying, by the one or more network-based devices and based at least in part on data included in the request, a user account associated with the first device;

identifying, by the one or more network-based devices and based at least in part on the data included in the request, a secondary device associated with the user account to perform the operation;

identifying, from a plurality of device drivers, a device driver associated with the secondary device, the device driver stored in one or more network-devices remote from the environment;

sending information indicative of the operation to the device driver;

causing the device driver to execute to generate a command configured to cause the secondary device to perform the operation;

receiving the command from the device driver;

identifying a communication protocol utilized by the secondary device;

determining that a third device, associated with the user account and residing in the environment, supports the communication protocol; and

sending, based at least in part on the determining, the command to one or more of the first device or the third device for the third device to issue the command via the communication protocol to the secondary device to cause the secondary device to perform the operation.

4. The method as recited in claim 3, wherein the secondary device comprises a light, a television, an audio system, a lock, a garage door opener, a washing machine, a dryer, a dishwasher, a coffee maker, a refrigerator, a door, or motorized window shades.

9. The method as recited in claim 3, further comprising installing an update to the device driver stored at the one or more network-based devices remote from the environment.

10. The method as recited in claim 3, further comprising:

receiving, from the first device, a request to register the secondary device with the user account; and

in response to receiving the request, storing an association between the secondary device and the user account.

11. The method as recited in claim 3, further comprising translating the request into the information indicative of the operation, wherein the information indicative of the request is actionable by the device driver to generate the command.

12. The method of claim 3, wherein:

the data includes an identifier which indicates at least one of the user account or the first device; and

identifying the user account is based at least in part on the identifier.

13. An apparatus comprising:

one or more network interfaces;

an input component;

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising;

receiving, from the input component, information indicative of a request to perform an operation on a secondary device;

sending the request to one or more network-based devices over the one or more network interfaces, the one or more network-based devices configured to identify the secondary device based at least in part on data in the request;

sending, to the one or more network-based devices, an identifier associated with at least one of the apparatus or a user account associated with the apparatus;

receiving, from the one or more network-based devices, a command generated by a device driver stored in the one or more network-based devices remote from an environment in which the apparatus resides and associated with the secondary device, the command to cause the secondary device to perform the operation;

identifying a communication protocol utilized by the secondary device;

determining that a third device, associated with the user account and residing in the environment, supports the communication protocol; and

sending, over the one or more network interfaces and based at least in part on the determining, the command to the third device to issue the command to the secondary device to cause the secondary device to perform the operation.

14. The apparatus as recited in claim 13, wherein the one or more computer-readable media of the apparatus is free from storing at least a portion of the device driver configured to generate the command for causing the secondary device to perform the operation.

15. The apparatus as recited in claim 13, wherein the input component comprises a microphone, and the request comprises a voice command.

16. The apparatus as recited in claim 13, wherein the input component comprises a camera, and the request comprises a gesture.

17. The apparatus as recited in claim 13, wherein the one or more network interfaces comprise a short-range wireless communication interface, and the sending of the command is over the short-range wireless communication interface.

18. The apparatus as recited in claim 13, wherein the secondary device comprises a light, a television, an audio system, a lock, a garage door opener, a washing machine, a dryer, a dishwasher, a coffee maker, a refrigerator, a door, or motorized window shades.

19. The apparatus as recited in claim 13, further comprising a speaker to output at least one of an indication that the apparatus has received the request to perform the operation on the secondary device or an indication that the command to cause the secondary device to perform the operation has been or will be sent to the secondary device.

20. The apparatus as recited in claim 13, the acts further comprising sending an identifier of the apparatus to the one or more network-based devices, the one or more network-based devices to identify at least one of an address or a device type of the secondary device using the identifier of the apparatus and to identify the device driver associated with the secondary device using the at least one of the address or the device type of the secondary device.