Training models using voice tags

• US 10,163,437 B1
• Filed: 06/02/2016
• Issued: 12/25/2018
• Est. Priority Date: 06/02/2016
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: St. Angel, Lindo, Strom, Nikko, Mutagi, Rohan
• Primary Examiner(s): Zhu, Richard Z
• Application Number:
  US15/171,787
• Time in Patent Office: 936 Days
• Field of Search: None
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: G05B 2219/35453 Voice announcement, oral, speech input
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Techniques for training machine-learning algorithms with the aid of voice tags are described herein. An environment may include sensors configured to generate sensor data and devices configured to perform operations. Sensor data as well as indications of actions performed by devices within the environment may be collected over time and analyzed to identify one or more patterns. Over time, a model that includes an association between this sensor data and device actions may be created and trained such that one or more device actions may be automatically initiated in response to identifying sensor data matching the sensor data of the model. To aid in the training, a user may utter a predefined voice tag each time she performs a particular sequence of actions, with the voice tag indicating to the system that temporally proximate sensor data and device-activity data should be used to train a particular model.

Claims

1. A system comprising:

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising;

receiving, during a first time period, first sensor data from a first sensor within an environment indicating that a door within the environment has been opened;

receiving, during the first time period, second sensor data from a second sensor within the environment indicating that motion has been detected within the environment;

receiving, during the first time period, an indication that a first device within the environment has been powered on;

receiving, from a second device within the environment, an audio signal representing speech of a user, the speech including an indication that the user has arrived at the environment;

performing speech recognition on the audio signal to identify the speech;

identifying a model previously associated with the indication that the user has arrived at the environment, the model based at least in part on sensor data previously received from the environment;

inputting the first sensor data, the second sensor data, and the indication that the first device has been powered on into one or more machine-learning algorithms to train the model;

receiving, during a second time period, third sensor data from the first sensor indicating that the door within the environment has been opened;

receiving, during the second time period, fourth sensor data from the second sensor within the environment indicating that motion has been detected within the environment;

determining that the third sensor data and the fourth sensor data correspond to a class specified by the model; and

sending an instruction to power on the first device within the environment at least partly in response to determining that the third sensor data and the fourth sensor data correspond to the class.

2. A system as recited in claim 1, the acts further comprising:

receiving, during the first time period, a first signal from a mobile device that is associated with a profile of the user, the first signal identifying the mobile device; and

receiving, during the second time period, a second signal from the mobile device, the second signal identifying the mobile device,

and wherein sending the instruction to power on the first device is further in response to receiving the second signal identifying the mobile device.

3. A system as recited in claim 1, the acts further comprising, prior to sending the instruction, outputting, on one or more devices in the environment, a query regarding whether to power on the first device.

4. A system as recited in claim 1, the acts further comprising:

receiving, prior to the second time, a voice command requesting;

(i) to power on a third device, and (ii) to associate an act of powering on the third device with the model;

storing an association between the act of powering on the third device and the model; and

sending an instruction to power on the third device at least partly in response to determining that the third sensor data and the fourth sensor data correspond to the class.

5. A method comprising:

receiving information indicative of events occurring, in a first sequential order, in an environment during a first time period, the information including;

a first collection of sensor data received from one or more sensors in the environment, the first collection of sensor data captured during the first time period, and

an indication, received from a first device in the environment, of at least one action performed by the first device during the first time period;

receiving an audio signal generated by a second device disposed within the environment and separate from the first device, the audio signal representing an utterance made by a user during the first time period;

identifying a model associated with the utterance;

inputting at least the first collection of sensor data and the indication of the at least one action into one or more machine-learning algorithms to train the model based at least partly on the first sequential order, the model determining a correlation between at least a portion of the first collection of sensor data and the at least one action performed by the first device;

receiving, from the one or more sensors in the environment, a second collection of sensor data captured during a second time period later than the first time period;

determining, based at least partly on the first sequential order, that at least a portion of the second collection of sensor data corresponds to the model; and

based at least in part on determining that the at least the portion of the second collection of sensor data corresponds to the model, sending an instruction to the first device to cause the first device to perform the at least one action based at least in part on the correlation.

6. A method as recited in claim 5, wherein the indication comprises a first indication, the at least one action comprises a first at least one action, the utterance comprises a first utterance, and the audio signal comprises a first audio signal, and wherein the method further comprises:

receiving, from the one or more sensors in the environment, a third collection of sensor data associated with a third time;

receiving a second indication of at least a second action performed by a third device separate from the first device;

receiving a second audio signal generated by the second device, the second audio signal representing a second utterance;

identifying the model based at least in part on the second utterance; and

inputting at least the third collection of sensor data and the second indication of the second action into the one or more machine-learning algorithms to train the model.

7. A method as recited in claim 5, wherein the model is associated with a profile of the user that uttered the utterance, and wherein the method further comprises:

determining that a user associated with the second collection of sensor data is the user that uttered the utterance,

and wherein sending the instruction to the first device is further based at least in part on determining that the user associated with the second collection of sensor data is the user that uttered the utterance.

8. A method as recited in claim 5, wherein the model is a first model associated with a profile of the user that uttered the utterance, and wherein the method further comprises:

identifying a different profile of another user, the different profile being associated with a second model that is associated with the utterance;

determining a second action associated with the second model; and

outputting a suggestion that the user associate the second action with the first model.

9. A method as recited in claim 5,

wherein determining, based at least partly on the first sequential order, that the at least the portion of the second collection of sensor data corresponds to the model includes comparing the first sequential order with a second sequential order of events corresponding to the second collection of sensor data.

10. A method as recited in claim 9,

wherein determining, based at least partly on the first sequential order, that the at least the portion of the second collection of sensor data corresponds to the model further includes comparing a first elapsed time of the first time period with a second elapsed time of the second time period.

11. A method as recited in claim 5, wherein:

the one or more sensors are separate from the first device and the second device,

the at least one action is performed in the environment and comprises activation of the first device, and

the first collection of sensor data is indicative of the at least one action at least one of a movement or a sound.

12. A method as recited in claim 5, further comprising:

receiving, from the one or more sensors in the environment, a third collection of sensor data captured by the one or more sensors at a third time different from the first time period and the second time period;

receiving an indication of at least one additional action performed, at the third time, by the first device in the environment; and

creating the model based at least in part on the third collection of sensor data and the at least one additional action.

13. A method as recited in claim 5, further comprising:

determining, based at least in part on determining that the at least the portion of the second collection of sensor data corresponds to the model, that the model associates the at least the portion of the second collection of sensor data with the at least one action,

wherein sending the instruction to the device to cause the device to perform the at least one action is based at least in part on determining that the model associates the at least the portion of the second collection of sensor data with the at least one action.

14. A system comprising:

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed on the one or more processors, cause the one or more processors to perform acts comprising;

receiving information indicative of events occurring, in a first sequential order, in an environment during a first time period, the information including;

a first collection of sensor data received from one or more sensors in the environment, the first collection of sensor data captured during the first time period, and

an indication, received from a first device in the environment, of at least one action performed by the first device during the first time period;

receiving an audio signal generated by a second device disposed within the environment and separate from the first device, the audio signal representing an utterance made by a user during the first time period;

identifying a model associated with the utterance;

inputting at least the first collection of sensor data and the indication of the at least one action into one or more machine-learning algorithms to train the model based at least partly on the first sequential order, the model determining a correlation between at least a portion of the first collection of sensor data and the at least one action performed by the first device;

receiving, from the one or more sensors in the environment, a second collection of sensor data captured during a second time period later than the first time period;

determining, based at least partly on the first sequential order, that at least a portion of the second collection of sensor data corresponds to the model; and

based at least in part on determining that the at least the portion of the second collection of sensor data corresponds to the model, sending an instruction to the first device to cause the first device to perform the at least one action based at least in part on the correlation.

15. A system as recited in claim 14, wherein the indication comprises a first indication, the at least one action comprises a first at least one action, the utterance comprises a first utterance, and the audio signal comprises a first audio signal, and the acts further comprising:

receiving, from the one or more sensors in the environment, a third collection of sensor data associated with a third time;

receiving a second indication of at least a second action performed by a third device separate from the first device;

receiving a second audio signal generated by the second device, the second audio signal representing a second utterance;

identifying the model based at least in part on the second utterance; and

inputting at least the third collection of sensor data and the second indication of the second action into the one or more machine-learning algorithms to train the model.

16. A system as recited in claim 14, wherein the model is associated with a profile of the user that uttered the utterance, and the acts further comprising:

determining that a user associated with the second collection of sensor data is the user that uttered the utterance,

and wherein sending the instruction to the first device is further based at least in part on determining that the user associated with the second collection of sensor data is the user that uttered the utterance.

17. A system as recited in claim 14, wherein the model is a first model associated with a profile of the user that uttered the utterance, and the acts further comprising:

identifying a different profile of another user, the different profile being associated with a second model that is associated with the utterance;

determining a second action associated with the second model; and

outputting a suggestion that the user associate the second action with the model.

18. A system as recited in claim 14, the acts further comprising:

outputting a request to confirm that the at least one action performed by the first device is to be associated with the model;

receiving an indication that the at least one action performed by the first device is to be associated with the model; and

storing an indication that the at least one action is to be performed at least partly in response to identifying sensor data that corresponds to the model.

19. A system as recited in claim 14, wherein determining, based at least partly on the first sequential order, that the at least the portion of the second collection of sensor data corresponds to the model includes determining that a level of similarity between the second collection of sensor data and the first collection of sensor data is greater than or equal to a threshold similarity level.

20. A system as recited in claim 14, the acts further comprising:

identifying the user based at least in part on the first collection of sensor data; and

determining that the user is associated with the model,

and wherein inputting at least the first collection of sensor data and the indication of the at least one action into the one or more machine-learning algorithms to train the model is based at least in part on determining that the user is associated with the model.