CONTROLLING HARDWARE IN AN ENVIRONMENT

• US 20140142729A1
• Filed: 11/21/2012
• Published: 05/22/2014
• Est. Priority Date: 11/21/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew, Hodges, Stephen Edward, Lobb, Ken
• Granted Patent: US 9,740,187 B2
• Time in Patent Office: Days
• US Class Current: 700/90
• CPC Class Codes: G05B 19/045 using logic state machines, consisting only of a memory or a programmable logic device containing the logic for the controlled machine and in which the state of its outputs is dependent on the state of its inputs or part of its own output states, e.g. binary decision controllers, finite state controllers
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

An instruction-storage machine holds instructions that, when executed by a logic machine, cause the logic machine to find a human subject in depth data acquired with one or more depth cameras arranged to image an environment, and to compute an aspect of the human subject from the depth data. Based on the computed aspect, the logic machine determines a change to be made in the environment and actuates appropriate hardware in the environment to make the change.

Claims

1. An instruction-storage machine holding instructions that, when executed by a logic machine, cause the logic machine to:

find a human subject in depth data acquired with one or more depth cameras arranged to image an environment;

compute an aspect of the human subject from the depth data;

based on the computed aspect, determine a change to be made in the environment; and

actuate hardware in the environment to make the change.

2. The instruction-storage machine of claim 1 wherein the change is a change in lighting intensity.

3. The instruction-storage machine of claim 1 wherein the change is a change in temperature in a locus of the environment.

4. The instruction-storage machine of claim 1 wherein the change is a change in a position or locking state of a door or window in the environment.

5. The instruction-storage machine of claim 1 wherein the change includes a change in a target locus of an audio and/or video system.

6. The instruction-storage machine of claim 1 wherein the computed aspect is a gesture that the human subject makes intentionally, to signal the change to be made in the environment, and wherein the gesture imitates an operation enacted with a hardware controller to make a corresponding change.

7. The instruction-storage machine of claim 1 wherein determining the change to be made includes intuiting a change desired by the human subject based on the computed aspect.

8. The instruction-storage machine of claim 1 wherein the computed aspect includes a location of the human subject on a coordinate system.

9. The instruction-storage machine of claim 1 wherein the human subject is one of a plurality of human subjects found in the depth data, and wherein the computed aspect includes the location of the human subject relative to the other found human subjects.

10. The instruction-storage machine of claim 1 wherein the computed aspect includes a characterization of a size or shape of one or more physical features of the human subject.

11. The instruction-storage machine of claim 1 wherein the computed aspect includes an identity of the human subject determined from among a plurality of candidate human subjects.

12. The instruction-storage machine of claim 1 wherein the computed aspect includes one or more of a posture of the human subject and a gestural change in the posture of the human subject.

13. The instruction-storage machine of claim 1 wherein the change to be made in the environment is a first change based on a computed aspect of a first human subject, and wherein the instructions are further executable to:

determine a second, conflicting change to be made in the environment based on a computed aspect of a second human subject, wherein the hardware is actuated to make the first change only if the first human subject ranks higher than the second human subject according to a trumping heuristic established for the environment.

14. The instruction-storage machine of claim 1 wherein the computed aspect includes an action of the human subject as defined by a series of gestural changes in a posture of the human subject.

15. The instruction-storage machine of claim 1 wherein the computed aspect includes a behavior pattern of the human subject as defined by a series of actions of the human subject, the actions defined by a series of gestural changes in a posture of the human subject.

16. A system comprising:

a sensor interface configured to receive depth data from one or more depth cameras arranged to image an environment;

an instruction-storage machine holding instructions that, when executed by a logic machine, cause the logic machine to find a human subject in depth data acquired with the one or more depth cameras, to compute an aspect of the human subject from the depth data, and based on the resolved aspect, to determine a change to be made in the environment; and

an actuator interface configured to actuate hardware in the environment to make the change.

17. The system of claim 16 wherein the depth data is received from a plurality of depth cameras, which provide a corresponding plurality of fields of view of the environment, and wherein the instructions further cause the logic machine to track the human subject from one field of view to another.

18. The system of claim 16 wherein the sensor interface is further configured to receive audio data from a plurality of microphones arranged in the environment, and wherein the aspect of the human subject is computed further based on the audio data.

19. The system of claim 17 wherein the sensor interface is further configured to receive sensory data from at least one of a passive infrared sensor and a color-image sensor, and wherein the aspect of the human subject is computed further based on the sensory data.

20. A system comprising:

a sensor interface configured to receive depth data from a plurality of depth cameras, each depth camera providing a different field of view of an environment;

an instruction-storage machine holding instructions that, when executed by a logic machine, cause the logic machine to find a human subject in depth data acquired by the depth cameras, to identify the human subject from a plurality of candidate human subjects, to recognize a preference of the human subject, to apply a heuristic to rank the revealed preference against preferences of others revealed in the depth data, and based on the preference and rank, to determine a change to be made in the environment; and

an actuator interface configured to actuate hardware in the environment to make the change.