Determining a device state based on user presence detection

• US 9,678,559 B1
• Filed: 09/18/2015
• Issued: 06/13/2017
• Est. Priority Date: 09/18/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Devries, David William, Shearer, Stewart Robin, Chhetri, Amit Singh, Devyver, Michael Serge, Sorek, Noam, Leung, Omar Sze, Hatipoglu, Serkan, Baldwin, Leo Benedict
• Primary Examiner(s): Stoynov, Stefan
• Application Number:
  US14/859,008
• Time in Patent Office: 634 Days
• CPC Class Codes: G01P 15/097 by vibratory elements
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/3265 Power saving in display device
• CPC Class Codes: G06F 1/3287 by switching off individual functional units in the computer system
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A decision engine executing on an electronic device may determine, using sensor data captured by multiple sensors of the device, whether a user is present in an environment that includes the device. If the user is determined to be present in the environment, the device may transition from a first state to a second state. The first state may be a first power state of the device in which the device is powered off or an idle or dormant state in which the device is powered on but a display of the device is powered off. Correspondingly, the second state may be a second power state of the device in which the device and the display are powered on and content is being rendered on the display. If the decision engine cannot make a determination based on the sensor data, a context engine may adjudicate the user presence determination.

Claims

1. A method, comprising:

receiving, by a computer processor of an electronic device, first data indicative of audio signals that are present in a detection region that includes the electronic device, wherein the first data is received at a microphone of the electronic device over a first period of time;

receiving, by the computer processor, second data indicative of vibrations that are present in the detection region, wherein the vibrations are sensed by an accelerometer of the electronic device over the first period of time;

filtering, by the computer processor, the first data to remove first noise from the first data and generate first filtered data;

filtering, by the computer processor, the second data to remove second noise from the second data and generate second filtered data;

selecting, by the computer processor, a first set of data values from the first filtered data and a second set of data values from the second filtered data;

multiplying, by the computer processor, the second set of data values by a weighting factor to generate a set of weighted data values;

determining, by the computer processor, a first sensor data signature comprising the first set of data values and the set of weighted data values, wherein the first sensor data signature is indicative of a combined output from the microphone and the accelerometer over the first period of time;

determining, by the computer processor, a second sensor data signature indicative of user presence in the detection region during a second period of time that is prior to the first period of time, comprising,

determining historical data generated by the accelerometer over the second period of time;

determining that a touch event was detected at a display of the electronic device during the second period of time, the touch event confirming user presence in the detection region during the second period of time,

generating the second sensor data signature, wherein the second sensor data signature includes at least a portion of the historical data generated by the accelerometer over the second period of time;

determining, by the computer processor, that the first sensor data signature is within a threshold tolerance of the second sensor data signature;

determining, by the computer processor and based at least in part on the first sensor data signature being within the threshold tolerance of the second sensor data signature, user presence in the detection region during the first period of time;

determining, by the computer processor, that the electronic device is in a first power state, wherein a display of the electronic device is powered off in the first power state; and

causing, by the computer processor, the electronic device to transition from the first power state to the second power state, wherein the display powered on in the second power state.

2. The method of claim 1, further comprising:

determining, by the computer processor, a third sensor data signature comprising the first set of data values and the set of weighted data values;

determining, by the computer processor, that the third sensor data signature is not within the threshold tolerance of the second sensor data signature;

activating, by the computer processor, a camera of the electronic device;

receiving, by the computer processor, third data comprising image frames generated by the camera, the image frame capturing at least a portion of the detection region; and

selecting, by the computer processor, a third set of data values from the third data,

wherein determining the first sensor data signature comprises including the third set of data values in the first sensor data signature.

3. The method of claim 1, further comprising:

receiving, by the computer processor, third data comprising a first signal sensed by the microphone and a second signal sensed by the accelerometer, wherein the first signal and the second signal are sensed over a third period of time subsequent to the first period of time;

determining, by the computer processor, that a high-occupancy state is associated with the detection region by;

determining, by the computer processor, that the first signal exceeds a first threshold value, and

determining, by the computer processor, that the second signal exceeds a second threshold value; and

causing, by the computer processor, the electronic device to remain in the second power state.

4. The method of claim 1, wherein the historical data is first historical data, the audio signals are first audio signals, and the vibrations are first vibrations, the method further comprising:

receiving, by a computer processor of the electronic device, third data indicative of ambient noise detected at the microphone during a third period of time subsequent to the first period of time;

receiving, by the computer processor, fourth data indicative of ambient vibrations sensed by the accelerometer during the third period of time;

determining, by the computer processor, second historical data generated by at least one of the microphone or the accelerometer over a fourth period of time prior to the third period of time;

determining, by the computer processor, that no user presence verification event occurred during the fourth period of time;

determining, by the computer processor, a third sensor data signature comprising at least a portion of the third data and at least a portion of the fourth data;

determining, by the computer processor, that the third sensor data signature is within the threshold tolerance of a fourth sensor data signature that includes at least a portion of the second historical data;

determining, by the computer processor, that the fourth sensor data signature is stored in association with an indication of user absence from the detection region during the fourth period of time;

determining, by the computer processor, user absence from the detection region during the third period of time; and

causing, by the computer processor, the electronic device to transition from the second power state to the first power state.

5. A method, comprising:

receiving, by a computer processor of an electronic device, first sensor data captured by a first sensor of the electronic device over a first period of time, wherein a first amount of power is supplied to the first sensor to activate the first sensor and a first latency is associated with activation of the first sensor;

receiving, by the computer processor, second sensor data captured by a second sensor of the electronic device over the first period of time, wherein a second amount of power is supplied to the second sensor to activate the second sensor and a second latency is associated with activation of the second sensor, and wherein the second amount of power is greater than the first amount of power or the second latency is longer than the first latency;

determining, by the computer processor, a first sensor data signature comprising at least a portion of the first sensor data and at least a portion of the second sensor data;

determining a second sensor data signature indicative of user presence in the detection region during a second period of time that is prior to the first period of time, wherein the second sensor data signature includes historical data captured by at least one of the first sensor or the second sensor over the second period of time;

determining, by the computer processor, that the first sensor data signature is within a threshold tolerance of the second sensor data signature;

determining, by the computer processor, that the electronic device is in a first state; and

causing, by the computer processor and based at least in part on the first sensor data signature being within the threshold tolerance of the second sensor data signature, the electronic device to transition from the first state to a second state that is different from the first state.

6. The method of claim 5, further comprising:

filtering, by the computer processor, the first sensor data to remove first noise from the first sensor data and generate first filtered sensor data;

filtering, by the computer processor, the second sensor data to remove second noise from the second sensor data and generate second filtered sensor data;

determining, by the computer processor, that a user presence verification event occurred during the second period of time, the user presence verification event confirming user presence in the detection region during the second period of time;

selecting, by the computer processor, a first set of sensor data values from the first filtered sensor data and a second set of sensor data values from the second filtered sensor data; and

applying, by the computer processor, a weighting factor to the second set of sensor data values to generate a set of weighted sensor data values,

wherein determining the first sensor data signature comprises generating the first sensor data signature using the first set of sensor data values and the set of weighted sensor data values.

7. The method of claim 6, wherein the user presence verification event comprises at least one of a touch event detected at a touch sensor of the electronic device or audio data indicative of a predetermined word or phrase.

8. The method of claim 5, further comprising:

determining, by the computer processor, that a location of the electronic device has changed from a first location to a second location; and

modifying the weighting factor based at least in part on one or more characteristics associated with the second location, wherein the one or more characteristics comprises at least one of a vibration characteristic of a physical material associated with the location, an acoustic characteristic, or a wireless signal characteristic.

9. The method of claim 5, further comprising:

determining, by the computer processor, that the first sensor data and the second sensor data include an insufficient amount of data to determine the user presence in the detection region during the first period of time; and

receiving, by the computer processor, third sensor data captured by a third sensor of the electronic device;

wherein determining the first sensor data signature comprises generating the first sensor signature further using at least a portion of the third sensor data.

10. The method of claim 5, wherein the historical data is first historical data, the method further comprising:

receiving, by the computer processor, third sensor data captured by the first sensor during a third period of time subsequent to the first period of time;

receiving, by the computer processor, fourth data captured by the second sensor during the third period of time;

determining, by the computer processor, second historical data generated by at least one of the first sensor or the second sensor over a fourth period of time prior to the third period of time;

determining, by the computer processor, that no user presence verification event occurred during the fourth period of time;

determining, by the computer processor, a third sensor data signature comprising at least a portion of the third data and at least a portion of the fourth data;

determining, by the computer processor, that the third sensor data signature is within the threshold tolerance of a fourth sensor data signature that includes at least a portion of the second historical data;

determining, by the computer processor, that the fourth sensor data signature is stored in association with an indication of user absence from the detection region during the fourth period of time;

determining, by the computer processor, user absence from the detection region during the third period of time; and

causing, by the computer processor, the electronic device to transition from the second state to the first state.

11. The method of claim 10, further comprising:

obtaining, by the computer processor, the second historical data from a remote server, wherein the second historical data further comprises data generated by one or more other sensors of one or more other electronic devices.

12. The method of claim 5, further comprising:

receiving, by the computer processor, third sensor data comprising additional sensor data captured by the first sensor during a third period of time subsequent to the first period of time and additional sensor data captured by the second sensor during the third period of time;

determining, by the computer processor, that the third sensor data is insufficient to determine user presence in the detection region during the third period of time;

determining, by the computer processor, occurrence of a user presence verification event during the third period of time, wherein the user presence verification event indicates the user presence in the detection region during the third period of time;

determining, by the computer processor, a predetermined association between the second power state and the user presence verification event; and

causing, by the computer processor, the electronic device to remain in the second power state.

13. The method of claim 5, further comprising:

determining, by the computer processor, that the first sensor data satisfies a first threshold confidence value; and

sequentially activating, by the computer processor, the second sensor based at least in part on the first sensor data satisfying the first threshold value, wherein the second sensor captures the second sensor data subsequent to the first sensor capturing the first sensor data.

14. An electronic device, comprising:

a display;

a plurality of sensors comprising a first sensor and a second sensor;

at least one memory storing computer-executable instructions; and

at least one processor operatively coupled to the display and the at least one memory, wherein the at least one processor is configured to execute the computer-executable instructions to;

receive first sensor data captured by the first sensor of the electronic device over a first period of time, wherein a first amount of power is supplied to the first sensor to activate the first sensor and a first latency is associated with activation of the first sensor;

receive second sensor data captured by the second sensor of the electronic device over the first period of time, wherein a second amount of power is supplied to the second sensor to activate the second sensor and a second latency is associated with activation of the second sensor, and wherein the second amount of power is greater than the first amount of power or the second latency is longer than the first latency;

determine a first sensor data signature comprising at least a portion of the first sensor data and at least a portion of the second sensor data;

determining a second sensor data signature indicative of user presence during a second period of time that is prior to the first period of time, wherein the second sensor data signature includes historical data captured by at least one of the first sensor or the second sensor over the second period of time;

determine that the first sensor data signature is within a threshold tolerance of the second sensor data signature;

determine that the electronic device is in a first state; and

cause, based at least in part on the determination the first sensor data signature being within the threshold tolerance of the second sensor data signature, the electronic device to transition from the first state to a second state that is different from the first state.

15. The electronic device of claim 14, wherein the at least one processor is further configured to execute the computer-executable instructions to:

filter the first sensor data to remove first noise from the first sensor data and generate first filtered sensor data;

filter the second sensor data to remove second noise from the second sensor data and generate second filtered sensor data;

determine that a user presence verification event occurred during the second period of time, the user presence verification event confirming user presence in the detection region during the second period of time;

select a first set of sensor data values from the first filtered sensor data and a second set of sensor data values from the second filtered sensor data; and

apply a weighting factor to the second set of sensor data values to generate a set of weighted sensor data values,

wherein the at least one processor is configured to determine the first sensor data signature by executing the computer-executable instructions to generate the first sensor data signature using the first set of sensor data values and the set of weighted sensor data values.

16. The electronic device of claim 14, wherein the at least one processor is further configured to execute the computer-executable instructions to:

determine that a location of the electronic device has changed from a first location to a second location; and

modify the weighting factor based at least in part on one or more characteristics associated with the second location, wherein the one or more characteristics comprises at least one of a vibration characteristic, an acoustic characteristic, or a wireless signal characteristic.

17. The electronic device of claim 14, wherein the at least one processor is further configured to execute the computer-executable instructions to:

determine that the first sensor data and the second sensor data include an insufficient amount of data to determine the user presence in the detection region during the first period of time; and

receive third sensor data captured by a third sensor of the plurality of sensors,

wherein the at least one processor is configured to determine the first sensor data signature by executing the computer-executable instructions to generate the first sensor signature further using at least a portion of the third sensor data.

18. The electronic device of claim 16, wherein the historical data is first historical data, and wherein the at least one processor is further configured to execute the computer-executable instructions to:

receive third sensor data captured by the first sensor during a third period of time subsequent to the first period of time;

receive fourth data captured by the second sensor during the third period of time;

determine second historical data generated by at least one of the first sensor or the second sensor over a fourth period of time prior to the third period of time;

determine that no user presence verification event occurred during the fourth period of time;

determine a third sensor data signature comprising at least a portion of the third data and at least a portion of the fourth data;

determine that the third sensor data signature is within the threshold tolerance of a fourth sensor data signature that includes at least a portion of the second historical data;

determine that the fourth sensor data signature is stored in association with an indication of user absence from the detection region during the fourth period of time;

determine user absence from the detection region during the third period of time; and

cause the electronic device to transition from the second state to the first state.

19. The electronic device of claim 14, wherein the at least one processor is further configured to execute the computer-executable instructions to:

receive third sensor data comprising additional sensor data captured by the first sensor during a third period of time subsequent to the first period of time and additional sensor data captured by the second sensor during the third period of time;

determine that the third sensor data is insufficient to determine user presence in the detection region during the third period of time;

determine occurrence of a user presence verification event during the third period of time, wherein the user presence verification event indicates the user presence in the detection region during the third period of time;

determine a predetermined association between the second power state and the user presence verification event; and

cause the electronic device to remain in the second power state.

20. The electronic device of claim 14, wherein the at least one processor is further configured to execute the computer-executable instructions to:

determine that the first sensor data satisfies a first threshold confidence value; and

sequentially activate the second sensor based at least in part on the first sensor data satisfying the first threshold value, wherein the second sensor captures the second sensor data subsequent to the first sensor capturing the first sensor data.