Gesture detection based on joint skipping

• US 8,633,890 B2
• Filed: 02/16/2010
• Issued: 01/21/2014
• Est. Priority Date: 02/16/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Tossell, Philip, Wilson, Andrew
• Primary Examiner(s): Lee, Gene W
• Application Number:
  US12/706,585
• Publication Number:
  US 20110199291A1
• Time in Patent Office: 1,435 Days
• Field of Search: 345156-158, 463 36- 39, 715/863
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training

Abstract

A system is disclosed for detecting or confirming gestures performed by a user by identifying a vector formed by non-adjacent joints and identifying the angle the vector forms with a reference point. Thus, the system skips one or more intermediate joints between an end joint and a proximal joint closer to the body core of a user. Skipping one or more intermediate joints results in a more reliable indication of the position or movement performed by the user, and consequently a more reliable indication of a given gesture.

Claims

1. In a system comprising a computing environment coupled to a capture device for capturing user position, a method of generating pose information for use in determining whether a user has performed a given gesture, the method comprising:

(a) detecting via the capture device a location of a first skeletal joint of the user position;

(b) detecting via the capture device a location of a second skeletal joint of the user position that is not a joint adjacent to the first skeletal joint; and

(c) generating the pose information, directly including the location of the first skeletal joint relative the location of the second skeletal joint, for use in determining whether a user has performed a given gesture;

wherein joint adjacency is determined along a user'"'"'s body.

2. The method of claim 1, said step (a) of detecting a location of a first skeletal joint comprising the step of detecting a location of an end joint.

3. The method of claim 1, said step (b) of detecting a location of a second skeletal joint that is not adjacent to the first skeletal joint comprising the step of detecting a location of a second skeletal joint that is two joints away from the first skeletal joint.

4. The method of claim 1, said step (b) of detecting a location of a second skeletal joint that is not adjacent to the first skeletal joint comprising the step of detecting a location of a second skeletal joint that is three joints away from the first skeletal joint.

5. The method of claim 1, said step (c) of generating pose information including the relative locations of the first and second skeletal joints comprising the step of generating a non-adjacent joint position vector forming an angle with a predefined reference.

6. The method of claim 1, further comprising:

(d) detecting a location of a third joint; and

(e) generating the pose information, including the location of the first skeletal joint relative the location of the third skeletal joint.

7. The method of claim 6, said step (d) of detecting a location of the third joint comprises the step of detecting the third joint at a location that is adjacent to the first joint.

8. The method of claim 6, said step (d) of detecting a location of the third joint comprises the step of detecting the third joint at a location that is spaced two or more joints away from the first joint.

9. A method performed by a computing environment of detecting whether a user has performed a given gesture, the method comprising:

(a) detecting a location of a first skeletal joint of the user position;

(b) detecting a location of a second skeletal joint of the user position that is not a joint adjacent to the first skeletal joint;

(c) generating a non-adjacent joint position vector having as end points the locations of the first and second skeletal joints detected in steps (a) and (b); and

(d) using the non-adjacent joint position vector generated in said step (c) to determine whether or not the user has performed a given gesture;

wherein joint adjacency is determined along a user'"'"'s body.

10. The method of claim 9, said step (a) of detecting a location of a first skeletal joint comprising the step of detecting a location of an end joint.

11. The method of claim 9, further comprising:

(e) detecting a third location of third skeletal joint of the user position that is not a joint adjacent to the first skeletal joint;

(f) generating a non-adjacent joint position vector having as end points the locations of the first and third skeletal joints detected in steps (a) and (e).

12. The method of claim 9, further comprising:

(g) detecting one or more additional locations of every other joint, beside the second joint, that is proximal of the first joint up to and including a joint located at a core of the user'"'"'s body that is at least two joints away from the first joint;

(h) generating additional non-adjacent joint position vectors, one each for each additional location detected in said step (g), each additional non-adjacent joint position vector extending between an additional location detected in said step (g) and the location of the first joint.

13. The method of claim 12, further comprising the step of using one or more of the additional non-adjacent joint position vectors generated in said step (e) in determine whether or not the user has performed a given gesture.

14. The method of claim 12, further comprising:

(j) detecting a location of an adjacent joint that is adjacent to the first joint; and

(k) generating an adjacent joint position vector extending between the location of the adjacent joint detected in said step (j) and the location of the first joint.

15. The method of claim 9, said steps (a) through (c) generating non-adjacent joint position vectors extending between the user'"'"'s ankle joint and hip joint.

16. The method of claim 9, said steps (a) through (c) generating non-adjacent joint position vectors extending between the user'"'"'s wrist joint and shoulder joint.

17. A system for detecting gestures performed by a user in real world space, the system comprising:

a capture device for capturing a depth image of a user within the field of view;

a computing environment for receiving the depth image from the capture device and determining a location of a plurality of joints, the plurality of joints including an end joint, at least one intermediate joint proximal of the end joint, and a core body joint proximal of the at least one intermediate joint; and

a processor within one of the capture device and computing environment for generating a non-adjacent joint position vector having end points at an end joint and a joint that is not adjacent to the end joint from one of the at least one intermediate joints and core body joint, the non-adjacent joint position vector used to determine whether the user has performed a predefined gesture;

wherein joint adjacency is determined along a user'"'"'s body.

18. The system of claim 17, the non-adjacent joint position vector comprising a first non-adjacent joint position vector, the processor further generating a second non-adjacent position vector having as end points the end joint and another joint not being a joint adjacent the end joint and not being a joint forming part of the first non-adjacent joint position vector.

19. The system of claim 17, the processor further generating an adjacent position vector having as end points the end joint and a joint adjacent the end point.

20. The system of claim 17, wherein the processor is included in the computing environment, the processor further generating pose information including the non-adjacent joint position vector, comparing the pose information against stored gestures, and determining whether the pose information indicates that a given gesture of the stored gestures has been performed.