Presence sensing

• US 10,372,191 B2
• Filed: 05/11/2012
• Issued: 08/06/2019
• Est. Priority Date: 05/12/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Valko, Edward Allen, Waldon, Matt, van der Merwe, Rudolph, Vieta, William Matthew, Haggerty, Myra, Nelson, Alex T., Jenkins, Karen Louise, Smith, Scott Thomas, Solem, Jan Erik
• Primary Examiner(s): Faragalla, Michael A
• Assistant Examiner(s): Bibbee, Chayce R
• Application Number:
  US13/469,996
• Publication Number:
  US 20120287035A1
• Time in Patent Office: 2,643 Days
• Field of Search: 345156, 713310, 713320-324, 713330, 713340, 340 58, 340 581- 585
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

One embodiment may take the form of a method of operating a computing device in a reduced power state and collecting a first set of data from at least one sensor. Based on the first set of data, the computing device determines a probability that an object is within a threshold distance of the computing device and, if so, the device activates at least one secondary sensor to collect a second set of data. Based on the second set of data, the device determines if the object is a person. If it is a person, a position of the person relative to the computing device is determined and the computing device changes its state based on the position of the person. If the object is not a person, the computing device remains in a reduced power state.

Claims

1. A method for determining whether a user is facing a computing device, the method comprising:

capturing an initial image using an image sensor having a first resolution;

computing, using a processor, a body detection parameter based on one or more macro features of a human body detected in the captured initial image, wherein the one or more macro features of a human body include at least one of an arm, a leg, a head, or a torso;

utilizing the body detection parameter to determine a first observation likelihood that the user is facing the computing device;

concurrent with capturing the initial image, capturing data from a second sensor;

using the data captured by the second sensor to compute a second observation likelihood that the user is facing the computing device;

using the first and the second observation likelihoods to determine a probability that the user is facing the computing device;

when the determined probability that the user is facing the computing device exceeds a first threshold, updating a state representation maintained by the computing device;

capturing one or more subsequent images with a camera having a second resolution higher than the first resolution of the image sensor;

using the captured one or more subsequent images to determine a facial feature detection parameter and a movement detection parameter; and

using the facial feature detection parameter and the movement detection parameter to update the probability that the user is facing the computing device;

determining that the updated probability that the user is facing the computing device exceeds a second threshold; and

further updating the state representation.

2. The method of claim 1, wherein:

the second sensor is a user input device;

the data captured from the second sensor is a signal from the user input device; and

the second observation likelihood is weighted more than the first observation likelihood in the determination of the probability that the user is facing the computing device.

3. The method of claim 1, wherein the state representation includes information regarding a three-dimensional position of the user with respect to the computing device and a velocity of the user with respect to the computing device.

24. The method of claim 3, wherein updating the probability that the user is facing the computing device uses a priori estimates for the three-dimensional position and the velocity of the user.

4. A computing device comprising:

an image sensor configured to capture image data at a first resolution;

at least one input device configured to capture additional data;

a processor coupled to the image sensor and to the at least one input device, the processor configured to;

process the image data to determine a first observation likelihood that a user is facing the computing device by computing a body presence parameter based on one or more macro features of a human body detected in the image data, wherein the one or more macro features of a human body include at least one of an arm, a leg, a head or a torso;

process the additional data to determine a second observation likelihood that the user is facing the computing device;

generate a predicted state representation, wherein the predicted state representation comprises an estimate for a position of the user and an estimate for a velocity of the user;

combine the predicted state representation and the first and the second observation likelihoods to determine a probability that the user is facing the computing device; and

when the probability that the user is facing the computing device exceeds a first threshold, change a state of the computing device;

capture additional image data with a camera having a second resolution higher than the first resolution;

use the captured additional image data to determine a movement parameter and a facial feature detection parameter;

update the probability that the user is facing the computing device using the body presence parameter, the movement parameter, and the facial feature detection parameter;

determine that the updated probability that the user is facing the computing device exceeds a second threshold; and

update the predicted state representation.

5. The computing device of claim 4, wherein the at least one input device comprises at least one of:

an additional image sensor;

a keyboard;

a touch screen;

a trackpad;

a microphone;

or

a mouse.

6. The computing device of claim 5, wherein the image sensor and the at least one input device operate concurrently.

7. The computing device of claim 4, wherein:

the at least one input device comprises an additional image sensor; and

at least one of the image sensor or the additional image sensor comprises a depth sensor.

8. The computing device of claim 7, wherein the image sensor and the additional image sensor operate concurrently.

9. The computing device of claim 4, wherein the change of state of the computing device comprises entering a warm-up state.

10. The computing device of claim 4, wherein the change of state comprises disabling a screen dimming function of the computing device.

11. The computing device of claim 4, wherein the movement parameter includes a velocity parameter.

12. The computing device of claim 4, wherein generating the predicted state representation uses a motion model comprising a state evolution expression that models movement of the user.

13. The computing device of claim 4, wherein the predicted state representation is updated using the probability that the user is facing the computing device.

14. A method of operating a computing device to provide user focus based functionality, the method comprising:

receiving 3-dimensional (3D) image data acquired by a 3D camera;

using the 3D image data to extract a first feature from a set of macro body features related to a user in local proximity to the computing device, wherein the set of macro body features related to the user includes at least one of an arm, a leg, a torso or a head;

concurrent with receiving the 3D image data, receiving sensor data from a second sensor;

separate from the 3D camera;

using the sensor data to extract a second feature from the set of macro body features related to the user being in local proximity to the computing device, the second feature being different from the first feature;

receiving a state representation comprising information related to the user being in local proximity to the computing device, wherein the information includes an estimated position of the user in 3D space and an estimated velocity of the user;

combining the state representation with a first observation likelihood based on the first feature and a second observation likelihood based on the second feature to;

produce a probability of the user facing the computing device; and

update the information in the state representation; and

changing a state of the computing device based on the probability of the user facing the computing device.

15. The method of claim 14, wherein:

the second sensor comprises a video camera;

the second sensor data comprises a frame of a video acquired by the video camera; and

the second feature extracted from the frame comprises at least one of;

a 3D blob;

a face;

a body;

or

a motion vector.

16. The method of claim 14, wherein the state representation further comprises a measurement model that assigns a first weight to the first extracted feature and assigns a second weight to the second extracted feature.

17. The method of claim 16, wherein the second weight is greater than the first weight when the first sensor data is noisier than the second sensor data.

18. The method of claim 14, wherein the estimated position of the user in 3D space and the estimated velocity of the user are based on a motion model that uses a state evolution expression to model a motion of the user.

19. The method of claim 14, further comprising updating the state representation using the first observation likelihood and the second observation likelihood.

20. The method of claim 19, wherein updating the state representation includes updating the estimated position of the user and the estimated velocity of the user.

21. The method of claim 19, wherein updating the state representation uses recursive Bayesian estimation.

22. The method of claim 14, wherein when the probability of the user facing the computing device exceeds a first threshold, changing the state of the computing device comprises at least one of:

changing a displayed screen saver image;

transitioning a display to an awake state;

transitioning the computing device to the awake state;

reducing a number of processes when transitioning the computing device to the awake state;

or

maintaining the computing device in an active state by modifying at least one timeout.

23. The method of claim 14, wherein when the probability of the user facing the computing device is less than a second threshold, changing the state of the computing device comprises at least one of:

changing an output on a display screen;

transitioning a display into a sleep mode;

or

transitioning the computing device to a sleep mode.