Methods for controlling electronic devices using gestures

• US 9,628,843 B2
• Filed: 11/21/2011
• Issued: 04/18/2017
• Est. Priority Date: 11/21/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Katz, Sagi, Adler, Avishai, Yahav, Giora
• Primary Examiner(s): Booker, Kelvin
• Application Number:
  US13/301,425
• Publication Number:
  US 20130131836A1
• Time in Patent Office: 1,975 Days
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 21/422 Input-only peripherals , i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] input devices also receiving signals from specially adapted client devices H04N21/4104
• CPC Class Codes: H04N 21/42221 Transmission circuitry, e.g. infrared [IR] or radio frequency [RF]
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45

Abstract

A system for controlling infrared (IR) enabled devices by projecting coded IR pulses from an active illumination depth camera is described. In some embodiments, a gesture recognition system includes an active illumination depth camera such as a depth camera that utilizes time-of-flight (TOF) or structured light techniques for obtaining depth information. The gesture recognition system may detect the performance of a particular gesture associated with a particular electronic device, determine a set of device instructions in response to detecting the particular gesture, and transmit the set of device instructions to the particular electronic device utilizing coded IR pulses. The coded IR pulses may imitate the IR pulses associated with a remote control protocol. In some cases, the coded IR pulses transmitted may also be used by the active illumination depth camera for determining depth information.

Claims

1. A method for controlling one or more electronic devices, comprising:

detecting a particular gesture;

determining a particular electronic device of the one or more electronic devices based on the particular gesture;

determining a plurality of coded light pulses for controlling the particular electronic device based on the particular gesture;

embedding a plurality of higher frequency light pulses within a first coded light pulse of the plurality of coded light pulses for controlling the particular electronic device;

projecting a sequence of light patterns originating from an active illumination depth camera into an environment, the sequence of light patterns includes the plurality of coded light pulses for controlling the particular electronic device; and

generating depth information associated with the environment, the generating depth information includes capturing a first image of the environment associated with a first high frequency light pulse of the plurality of higher frequency light pulses and generating the depth information using the first image, the generating depth information includes capturing one or more images associated with reflections of at least a subset of the sequence of light patterns within the environment.

2. The method of claim 1, wherein:

the detecting the particular gesture includes performing skeletal tracking of a person within the environment performing the particular gesture.

3. The method of claim 1, further comprising:

determining the particular electronic device of the one or more electronic devices based on a direction of motion associated with the particular gesture.

4. The method of claim 1, wherein:

the active illumination depth camera utilizes time-of-flight techniques for obtaining the depth information; and

the plurality of coded light pulses correspond with a remote control protocol.

5. The method of claim 1, wherein:

the generating depth information overlaps in time with the projecting a sequence of light patterns.

6. The method of claim 1, further comprising:

detecting a new end user command based on the first image, the sequence of light patterns includes one or more new light pulses associated with the new end user command.

7. The method of claim 1, further comprising:

determining environmental characteristics associated with the environment; and

modifying a light intensity associated with one or more light patterns of the sequence of light patterns based on the environmental characteristics.

8. The method of claim 1, wherein:

the determining a plurality of coded light pulses includes acquiring one or more IR transmission protocols associated with the one or more electronic devices.

9. The method of claim 1, wherein:

the determining a plurality of coded light pulses includes automatically identifying the one or more electronic devices.

10. The method of claim 9, wherein:

the automatically identifying the one or more electronic devices includes transmitting one or more commands associated with the particular electronic device, the one or more commands cause the particular electronic device to generate sound, the automatically identifying the one or more electronic devices includes detecting and localizing sounds within the environment in response to the transmitting one or more commands.

11. A system for controlling one or more electronic devices, comprising:

an active illumination depth camera; and

one or more processors configured to detect a particular gesture and determine a particular electronic device of the one or more electronic devices based on the particular gesture, the one or more processors configured to determine a plurality of coded IR pulses for communicating with the particular electronic device based on the particular gesture and embed a plurality of higher frequency light pulses within a first coded light pulse of the plurality of coded IR pulses, the one or more processors configured to direct the active illumination depth camera to project a sequence of light patterns into an environment, the sequence of light patterns includes the plurality of coded IR pulses for communicating with the particular electronic device, the active illumination depth camera configured to capture a first image of the environment associated with a first high frequency light pulse of the plurality of higher frequency light pulses, the one or more processors configured to generate depth information associated with the environment based on the first image, the one or more processors configured to acquire one or more images associated with reflections of at least a subset of the sequence of light patterns within the environment and generate the depth information using the one or more images.

12. The electronic device of claim 11, wherein:

the one or more processors configured to perform skeletal tracking of a person within the environment and detect the particular gesture based on the skeletal tracking.

13. One or more storage devices containing non-transitory processor readable code for programming one or more processors to perform a method for controlling one or more electronic devices comprising the steps of:

identifying a particular gesture associated with the one or more electronic devices;

determining a particular electronic device of the one or more electronic devices based on a direction of motion associated with the particular gesture;

determining a plurality of coded IR pulses for controlling the particular electronic device;

embedding a plurality of higher frequency light pulses within a first coded light pulse of the plurality of coded IR pulses for controlling the particular electronic device;

modulating one or more light patterns projected from an active illumination depth camera into an environment, the one or more light patterns includes the plurality of coded IR pulses for controlling the particular electronic device; and

generating depth information associated with the environment, the generating depth information includes capturing a first image of the environment associated with a first high frequency light pulse of the plurality of higher frequency light pulses and generating the depth information using the first image, the generating depth information includes capturing one or more images associated with reflections of at least a subset of the one or more light patterns within the environment.

14. The one or more storage devices of claim 13, wherein:

the modulating one or more light patterns includes modulating an amplitude of at least a subset of the one or more light patterns projected from the active illumination depth camera.

15. The one or more storage devices of claim 13, wherein:

the modulating one or more light patterns includes modulating a frequency of the one or more light patterns projected from the active illumination depth camera.

16. The one or more storage devices of claim 13, further comprising:

identifying an end user command corresponding with the particular gesture and determining the plurality of coded IR pulses based on the end user command.