Method and system implementing user-centric gesture control

• US 9,569,005 B2
• Filed: 04/03/2014
• Issued: 02/14/2017
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ahmed, Abdelrehim, Rafii, Abbas, Tracy, Colin
• Primary Examiner(s): Chang, Jon
• Application Number:
  US14/244,151
• Publication Number:
  US 20140215407A1
• Time in Patent Office: 1,048 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A user-centric method and system to identify user-made gestures to control a remote device images the user using a three-dimensional image system, and defines at least one user-centric three-dimensional detection zone dynamically sized appropriately for the user, who is free to move about. Images made within the detection zone are compared to a library of stored gestures, and the thus identified gesture is mapped to an appropriate control command signal coupleable to the remote device. The method and system also provides of a first user to hand off control of the remote device to a second user.

Claims

1. A machine system implemented method providing user-centric gesture recognition, the method comprising:

receiving three-dimensional (3D) tracking data representative of respective positionings of respective ones of plural user body parts of an automatically tracked first user of the machine system, the plural user body parts being in a field of sensing of one or more sensors of the machine system;

defining a first movable 3D frame of reference that moves in coordination with movement of a first subset of one or more of the plural user body parts of the tracked first user but not in coordination with movement of one or more other body parts;

defining a major interaction enabled zone positioned within the first movable 3D frame of reference and moving as the first movable 3D frame of reference moves;

determining based at least on the received 3D tracking data if at least one of the other body parts of the tracked first user is within the defined major interaction enabled zone;

based at least on the received 3D tracking data, recognizing a gesture made by the at least one of the other body parts that is determined to be within the defined major interaction enabled zone;

comparing said recognized gesture to contents of a gesture library of one or more pre-defined gestures that are useable to control by way of gesture, a gesture controllable device of the machine system; and

based at least on finding a match between a recognized said user-gesture and contents of said gesture library, causing a carrying out of a corresponding gesture initiated action by the gesture controllable device.

2. The method of claim 1, further including:

defining within the major interaction enabled zone two or more spaced apart, 3D hot zones that move as the major interaction enabled zone moves;

determining if at least one of the other body parts of the tracked first user is within at least one of the defined hot zones;

recognizing a gesture made by the at least one of the other body parts that is determined to be within at least one of the defined hot zones;

wherein said comparing of the recognized gesture to contents of the gesture library includes determining if the respectively recognized gesture includes at least one of positioning and moving of predefined body parts within at least one of the two or more spaced apart, 3D hot zones in accordance with one or more pre-defined gestures of the gesture library that call for the pre-defined gesture to be made respectively within at least one of the two or more spaced apart, 3D hot zones.

3. The method of claim 1, wherein said gesture library stores at least one gesture selected from a group consisting of (i) a gesture signifying desire to acquire control of said gesture controllable device, (ii) a gesture signifying handing off of gesture control of said gesture controllable device from said user to a successor user, (iii) a gesture made with only one hand, (iv) a gesture made with two hands, and (v) a gesture causing a menu to be displayed.

4. The method of claim 1, further including causing said machine system to alert the user that a user-gesture may not have been correctly recognized, and inviting said user to confirm a gesture recognized by the machine system.

5. The method of claim 4, wherein a user can confirm the gesture recognized by the machine system by making a user movement within a sub-region defined within said first movable 3D frame of reference.

6. The method of claim 5, further including causing said machine system to signal said user that improper recognition of a questionable gesture may have occurred, and causing said machine system to provide an option for said user to at least confirm said questionable gesture as being correctly recognized by the machine system.

7. The method of claim 1, wherein said gesture library stores gestures that are independent of previously made gestures.

8. The method of claim 1, wherein said one or more sensors of the machine system include image capture devices that are configured to provide depth image data by use of at least one of a (i) parallex-method imaging system, (ii) a time-of-flight imaging system, (iii) a phase-based time-of-flight imaging system, (iv) a shutter-based time-of-flight imaging system, and (v) a system using at least two spaced-apart cameras.

9. A machine system having user-concentric gesture recognition, the system including stored code configured to cause one or more programmable parts of the machine system to carry out a method comprising:

receiving three-dimensional (3D) image data representative of respective positionings of respective ones of plural user body parts of an automatically tracked first user of the machine system, the plural user body parts being in a field of view of one or more image capture devices of the machine system;

identifying the automatically tracked first user when said user is within said field of view;

defining at least a first movable 3D frame of reference that moves in coordination with movement of a first subset of one or more of the plural user body parts of the identified first user but not in coordination with movement of one or more other body parts;

defining a major interaction enabled zone positioned within the first movable 3D frame of reference, the defining of the major interaction enabled zone including sizing and positioning the major interaction enabled zone in accordance with parameters of an accessible body profile of the identified first user;

determining if at least one of the other user body parts of the identified first user not in the first subset of body parts, is within the defined major interaction enabled zone;

recognizing a gesture made by the at least one of the other body parts that is determined to be within the defined major interaction enabled zone;

comparing said recognized gesture to contents of a gesture library including a plurality pre-defined gestures defined as occurring within at least one subregion of the major interaction enabled zone, at least one of the pre-defined gestures being defined as useable to control a gesture controllable device of the machine system; and

based at least on finding a match between a recognized said user-gesture and contents of said gesture library, outputting a signal useable to control said gesture controllable device.

10. The machine system of claim 9, wherein the carried out method further includes defining two or more spaced apart subregions within said major interaction enabled zone wherein presence of a portion of said user must be detected in order for a gesture to be a recognized gesture.

11. The machine system of claim 9 wherein said gesture library includes at least one gesture selected from a group consisting of (i) a gesture signifying desire to acquire control of said gesture controllable device, (ii) a gesture signifying handing off control of said gesture controllable device from said user to a successor user, (iii) a gesture made with only one hand, (iv) a gesture made with two hands, and (v) a gesture causing a menu to be displayed.

12. The machine system of claim 9, wherein the carried out method further includes causing said machine system to alert a user that a user-gesture may not have been correctly recognized by the machine system.

13. The machine system of claim 12 wherein the carried out method further includes inviting said user to confirm improper recognition of a gesture.

14. The machine system of claim 13 wherein the carried out method further includes enabling a user to confirm improper recognition of a gesture by making a user movement within a sub-region defined within said major interaction enabled zone.

15. The machine system of claim 13, wherein the carried out method further includes outputting a signal causing said device to alert said user that improper recognition of a gesture may have occurred, and causing said device to provide an option for said user to at least confirm said improper gesture recognition.