State detection and responses for electronic devices

• US 10,580,407 B1
• Filed: 12/08/2017
• Issued: 03/03/2020
• Est. Priority Date: 12/08/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sriram, Vinay Kotikalapudi, Welbourne, William Evan, Patel, Jay, Eberhardt, Sven
• Primary Examiner(s): Patel, Shreyans A
• Application Number:
  US15/836,414
• Time in Patent Office: 816 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 12/12 Arrangements for remote connection or disconnection of substations or of equipment thereof
• CPC Class Codes: H04L 12/2803 Home automation networks
• CPC Class Codes: Y02D 30/50 in wire-line communication networks, e.g. low power modes or reduced link rate

Abstract

This disclosure describes, in part, techniques for utilizing global models to generate local models for electronic devices in an environment, and techniques for utilizing the global models and/or the local models to provide notifications that are based on anomalies detected within the environment. For instance, a remote system may receive an identifier associated with an electronic device and identify a global model using the identifier. The remote system may then receive data indicating state changes of the electronic device and use the data and the global model to generate a local model associated with the electronic device. Using the global model and/or local model, the remote system can identify anomalies associated with the electronic device and, in response to identifying an anomaly, notify the user. The remote system can further cause the electronic device to change states after receiving a request from the user.

Claims

1. A system comprising:

one or more processors; and

one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving identifier data associated with an electronic device that is physically located in an environment;

receiving state data representing a power state associated with the electronic device, wherein time data is associated with the state data, the time data indicating a first time that the electronic device was in the powered-on state;

associating, using the identifier data and the state data representing the power state, a global model with the electronic device, the global model indicating a second time that the electronic device is expected to be in a powered-on state;

generating a local model for the electronic device based at least in part on the global model and the state data representing the power state, the local model indicating a third time that the electronic device is expected to be in the powered-on state;

determining that the electronic device is in the powered-on state during a fourth time that the electronic device is expected to be in a powered-off state;

determining, based at least in part on determining that the electronic device is in the powered-on state during the fourth time that the electronic device is expected to be in the powered-off state, to output a query regarding whether to power off the electronic device;

sending, to a voice-controlled device, second audio data for output by the voice-controlled device, the second audio data representing the query;

receiving third audio data from the voice-controlled device, the third audio data representing an utterance;

determining, based at least in part on the third audio data, to cause the electronic device to transition from the powered-on state to the powered-off state; and

sending an instruction that causes the electronic device to transition from the powered-on state to the powered-off state.

2. The system as recited in claim 1, the operations further comprising, at least partly after the determining to output the query and at least partly prior to the sending of the second audio data:

storing a first indication that that the query is to be output during a subsequent user interaction with the voice-controlled device;

storing a second indication of an expiration time at which to remove the first indication;

receiving, at a time prior to the expiration time, a third indication of a subsequent user interaction with the voice-controlled device,

and wherein sending the second audio data comprises sending the second audio data representing the query to the voice-controlled device at least partly in response to receiving, at the time prior to the expiration time, the third indication of the subsequent user interaction with the voice-controlled device.

3. The system as recited in claim 1, wherein the global model is a first global model, and wherein associating the global model with the electronic device comprises:

identifying a type of device that is associated with the identifier data;

identifying a first global model that is associated with the type of device, the first global model indicating the second time that the electronic device is expected to be in a powered-on state;

identifying a second global model that is associated with the type of device, second global model indicating a fifth time that the electronic device is expected to be in the powered-on state;

determining, based at least in part on the state data, that the electronic device is in the powered-on state during the second time and in the powered-off state during the fifth time; and

associating the first global model with the electronic device based on the electronic device is in the powered-on state during the second time and in the powered-off state during the fifth time.

4. A method comprising:

generating a data structure associated with an electronic device, the data structure indicating at least a first time that the electronic device is expected to be in a first state and a second time that the electronic device is expected to be in a second state;

determining, at the second time, that the electronic device is currently in the first state;

determining, based at least in part on the electronic device being in the first state at the second time, to output a query regarding whether to cause the electronic device to transition from the first state to the second state; and

causing audio data to be sent to a voice-controlled device for output by the voice-controlled device, the audio data representing the query.

5. The method as recited in claim 4, further comprising:

receiving additional audio data from the voice-controlled device, the additional audio data representing an utterance;

determining, based at least in part on the utterance, to cause the electronic device to switch from the first state to the second state; and

sending an instruction that causes the electronic device to transition from the first state to the second state.

6. The method as recited in claim 5, further comprising generating one or more parameters of the data structure based at least in part on causing the electronic device to transition from the first state to the second state.

7. The method as recited in claim 4, further comprising:

receiving additional audio data from the voice-controlled device, the additional audio data representing an utterance;

based at least in part on the utterance, refraining from causing the electronic device to transition from the first state to the second state; and

generating one or more parameters of the data structure based at least in part on refraining from causing the electronic device to transition from the first state to the second state.

8. The method as recited in claim 4, wherein the data structure comprises a first data structure, and wherein the method further comprises:

receiving identifier data associated with the electronic device; and

identifying a second data structure based at least in part on the identifier data, the second data structure identifying at least a third time that the electronic device is expected to be in the first state and a fourth time that the electronic device is expected to be in the second state,

wherein generating the first data structure associated with the electronic device is based at least in part on the second data structure.

9. The method as recited in claim 4, further comprising:

receiving first data indicating a third time that the electronic device transitioned from the first state to the second state; and

receiving second data indicating a fourth time that the electronic device transitioned from the second state to the first state,

wherein generating the data structure is based at least in part on the first data and the second data.

10. The method as recited in claim 4, wherein the audio data is first audio data, and wherein the method further comprising:

receiving, from the voice-controlled device, second audio data representing a first request to transition the electronic device from the first state to the second state; and

receiving, from the voice-controlled device, third audio data representing a second request to transition the electronic device from the second state to the first state,

wherein generating the data structure is based at least in part on the second audio data and the third audio data.

11. The method as recited in claim 4, further comprising:

determining that the electronic device was transitioned from the second state to the first state at the first time; and

determining, using the data structure, that the electronic device is expected to be transitioned from the second state to the first state at the first time,

wherein determining to output query is further based at least in part on determining that the electronic device is expected to transition from the second state to the first state at the first time.

12. The method as recited in claim 4, further comprising:

determining a confidence value associated with at least one of the data structure or the second time; and

determining that the confidence value exceeds a threshold,

wherein determining to output the query is further based at least in part on determining that the confidence value exceeds the threshold.

13. The method as recited in claim 4, further comprising:

determining that a threshold amount of time has passed since sending the audio data to the voice-controlled device;

determining a type of device that is associated with the electronic device; and

sending, after the threshold amount of time, and based at least in part on the type of device, an instruction that causes the electronic device to transition from the first state to the second state.

14. The method as recited in claim 4, further comprising:

receiving, from the electronic device, data indicating that the electronic device is in the second state at the second time,

wherein determining, at the second time, that the electronic device is currently in the first state is based at least in part on the data.

15. The method as recited in claim 4, further comprising:

receiving, from the voice-controlled device, additional audio data representing an utterance;

sending, based at least in part on the additional audio data, a first instruction that causes the electronic device to transition from the second state to the first state; and

determining, at the second time, that a second instruction has yet to be sent that causes the electronic device to transition from the first state to the second state,

wherein determining, at the second time, that the electronic device is currently in the first state is based at least in part on determining that the second instruction has yet to be sent.

16. A system comprising:

one or more processors; and

one or more computer-readable media storing instructions that, when executed by the one or more processor, cause the one or more processors to perform operations comprising;

receiving identifier data representing an identifier associated with an electronic device;

receiving, from a voice-controlled device, first audio data representing a first request to transition the electronic device from a first state to a second state at a first time;

receiving, from the voice-controlled device, second audio data representing a second request to transition the electronic device from the second state to the first state at a second time;

identifying, based at least in part on the first data, the second data, and the third data, a first data structure to associate with the electronic device, the first data structure indicating at least a third time that the electronic device is expected to be in the first state and a fourth time that the electronic device is expected to be in the second state; and

generating a second data structure associated with the electronic device based at least in part on the first data structure, the second data, and the third data, the second data structure indicating at least a fifth time that the electronic device is expected to be in the first state and a sixth time that the electronic device is expected to be in the second state.

17. The system as recited in claim 16, the operations further comprising:

determining, using the identifier data, a type of device that is associated with the electronic device; and

determining, using the identifier data, a role associated with the electronic device,

wherein identifying the first data structure to associate with the electronic device comprises identifying the first data structure to associate with the electronic device based at least in part the type of device and the role.

18. The system as recited in claim 16, the operations further comprising:

determining, at the sixth time, that the electronic device is currently in the first state;

determining, based at least in part on the electronic device being in the first state at the sixth time, to output a query regarding whether to transition the electronic device to the second state;

sending, to the voice-controlled device, third audio data for output by the voice-controlled device, the third audio data representing the query;

receiving, from the voice-controlled device, fourth audio data representing a request to transition the electronic device from the first state to the second state;

sending an instruction that causes the electronic device to transition from the first state to the second state; and

generating one or more parameters for the second data structure based at least in part on the second audio data.

19. The system as recited in claim 16, the operations further comprising:

determining, at the fifth time, that the electronic device is currently in the second state;

determining, based at least in part on the electronic device being in the second state at the fifth time, to output a query regarding whether to transition the electronic device to the first state;

sending, to the voice-controlled device, third audio data for output by the voice-controlled device, the third audio data representing the query;

receiving, from the voice-controlled device, fourth audio data representing a request to transition the electronic device from the second state to the first state; and

sending an instruction that causes the electronic device to transition from the second state to the first state.

20. The system as recited in claim 16, the operations further comprising:

receiving, from the voice-controlled device, third audio data representing a third request to transition the electronic device from the first state to the second state at a seventh time;

receiving, from the voice-controlled device, fourth audio data representing a fourth request to transition the electronic device from the second state to the first state at a seventh time; and

generating one or more parameters for the second data structure based at least in part on the third audio data and the fourth audio data.