Controlling hardware in an environment

• US 9,740,187 B2
• Filed: 11/21/2012
• Issued: 08/22/2017
• Est. Priority Date: 11/21/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lobb, Ken, Wilson, Andrew, Hodges, Stephen Edward
• Primary Examiner(s): Karim, Ziaul
• Application Number:
  US13/683,989
• Publication Number:
  US 20140142729A1
• Time in Patent Office: 1,735 Days
• Field of Search: 700 90, 382103
• CPC Class Codes: G05B 19/045 using logic state machines, consisting only of a memory or a programmable logic device containing the logic for the controlled machine and in which the state of its outputs is dependent on the state of its inputs or part of its own output states, e.g. binary decision controllers, finite state controllers
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

An instruction-storage machine holds instructions that, when executed by a logic machine, cause the logic machine to find a human subject in depth data acquired with one or more depth cameras arranged to image an environment, and to compute an aspect of the human subject from the depth data. Based on the computed aspect, the logic machine determines a change to be made in the environment and actuates appropriate hardware in the environment to make the change.

Claims

1. A residential control system, comprising:

a controller operatively coupled to one or more hardware components and one or more depth cameras each installed in a residential building, the depth cameras being arranged to image an interior of the residential building, the controller including a logic machine and an instruction-storage machine holding instructions that, when executed by the logic machine, cause the logic machine to;

find a human subject in depth data acquired with the one or more depth cameras;

compute an aspect of the human subject from the depth data, the computed aspect including a characterization of a size or shape of one or more physical features of the human subject and a behavior pattern of the human subject as defined by a series of actions of the human subject, the actions defined by a series of gestural changes in a posture of the human subject;

based on the computed aspect, determine an environmental change to be made in the residential building; and

actuate the one or more hardware components to make the environmental change.

2. The residential control system of claim 1 wherein the environmental change is a change in lighting intensity in the residential building, and wherein the one or more hardware components includes a lamp.

3. The residential control system of claim 1 wherein the environmental change is a change in temperature in a locus of the residential building, and wherein the one or more hardware components includes a thermostat.

4. The residential control system of claim 1 wherein the environmental change is a change in a position or locking state of a door or window in the residential building, and wherein the one or more hardware components includes a lock.

5. The residential control system of claim 1 wherein the environmental change includes a change in a target locus of audio and/or video, and wherein the one or more hardware components includes an audio/video system.

6. The residential control system of claim 1 wherein the computed aspect includes a gesture that the human subject makes intentionally, to signal the environmental change to be made in the residential building, and wherein the gesture imitates an operation enacted by a hardware controller to make the environmental change.

7. The residential control system of claim 1 wherein determining the environmental change to be made includes intuiting a change desired by the human subject based on the computed aspect.

8. The residential control system of claim 1 wherein the computed aspect reveals a location of the human subject on a coordinate system in the residential building.

9. The residential control system of claim 1 wherein the human subject is one of a plurality of human subjects found in the depth data, and wherein the computed aspect reveals a location of the human subject relative to the other found human subjects.

10. The residential control system of claim 1 wherein the computed aspect includes an identity of the human subject determined from among a plurality of candidate human subjects.

11. The residential control system of claim 1 wherein the environmental change to be made in the residential building is a first change based on a computed aspect of a first human subject, and wherein the instructions are further executable to:

determine a second, conflicting change to be made in the residential building based on a computed aspect of a second human subject, wherein the one or more hardware components are actuated to make the first change only if the first human subject ranks higher than the second human subject according to a trumping heuristic established for the residential building.

12. A residential control system comprising:

a sensor interface configured to receive depth data from a plurality of depth cameras installed in a residential building and arranged to image an interior of the residential building;

a controller operatively coupled to the sensor interface, the controller including an instruction-storage machine holding instructions that, when executed by a logic machine, cause the logic machine to find a human subject in depth data acquired with the plurality of depth cameras, to compute an aspect of the human subject from the depth data, and based on the computed aspect, to determine an environmental change to be made in the residential building, wherein the computed aspect includes an identity of the human subject determined from among a plurality of candidate human subjects and a behavior pattern of the human subject as defined by a series of actions of the human subject, the actions defined by a series of gestural changes in a posture of the human subject; and

an actuator interface operatively coupled to the controller and wirelessly coupled to one or more hardware components installed in the residential building and configured to actuate the one or more hardware components in order to make the environmental change.

13. The residential control system of claim 12 wherein the computed aspect includes a characterization of a size or shape of one or more physical features of the human subject.

14. The residential control system of claim 12 wherein the plurality of depth cameras provide a corresponding plurality of fields of view of the residential building, and wherein the instructions further cause the logic machine to track the human subject from one field of view to another.

15. The residential control system of claim 14 wherein the sensor interface is further configured to receive sensory data from at least one of a passive infrared sensor and a color-image sensor, and wherein the computed aspect of the human subject is computed further based on the sensory data.

16. The residential control system of claim 12 wherein the sensor interface is further configured to receive audio data from one or more microphones arranged in the residential building, and wherein the computed aspect of the human subject is computed further based on the audio data.

17. A residential control system comprising:

a sensor interface configured to receive depth data from a plurality of depth cameras installed in a residential building and arranged to image an interior of the residential building, each depth camera providing a different field of view of the residential building;

a controller operatively coupled to the sensor interface, the controller including an instruction-storage machine holding instructions that, when executed by a logic machine, cause the logic machine to find a human subject in depth data acquired by the depth cameras, to identify the human subject from a plurality of candidate human subjects, to recognize a preference of the human subject based on a computed aspect of the human subject, to apply a heuristic to rank the recognized preference against preferences of others recognized in the depth data, and based on the preference and rank, to determine an environmental change to be made in the residential building, wherein the computed aspect includes a characterization of a size or shape of one or more physical features of the human subject and a behavior pattern of the human subject as defined by a series of actions of the human subject, the actions defined by a series of gestural changes in a posture of the human subject; and

an actuator interface operatively coupled to the controller and configured to actuate one or more hardware components in the residential building in order to make the environmental change.