Gesture recognition system using depth perceptive sensors

• US 20110291926A1
• Filed: 03/04/2008
• Published: 12/01/2011
• Est. Priority Date: 02/15/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Surucu, Fahri, Tomasi, Carlo, Rafii, Abbas, Gokturk, Salih Burak
• Granted Patent: US 10,242,255 B2
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45

Abstract

Acquired three-dimensional positional information is used to identify user created gesture(s), which gesture(s) are classified to determine appropriate input(s) to an associated electronic device or devices. Preferably at at least one instance of a time interval, the posture of a portion of a user is recognized, based at least one factor such as shape, position, orientation, velocity. Posture over each of the instance(s) is recognized as a combined gesture. Because acquired information is three-dimensional, two gestures may occur simultaneously.

Claims

1. A method enabling user-interaction with an electronic device using at least one gesture made by a movable body part of said user, the method including the following steps:

(a) using a sensor system, operable absent ambient light, from which three-dimensional coordinate information is obtainable to acquire three-dimensional space positional information for a plurality of discrete regions on said body part relative to said sensor system;

(b) detecting from said three-dimensional positional information acquired at step (a) occurrence of said at least one gesture; and

(c) classifying each said gesture detected at step (b) as at least one input to said electronic device.

2. The method of claim 1, wherein at step (a), said sensor system is provided with an array of pixels to capture an image of a scene including said at least one body part, wherein each pixel in said array corresponds to a discrete region of said scene.

4. The method of claim 1, wherein at least one said gesture is classified as one of (i) a dynamic gesture, and (ii) a static gesture.

5. The method of claim 1, wherein:

at least a first gesture and a second gesture occur simultaneously;

step (b) detects each said gesture; and

step (c) classifies each said gesture;

wherein classification at step (c) includes at least one of (i) classifying said first gesture and said second gesture so as to comprise a single input to said device, and (ii) classifying said first gesture and said second gesture so as to comprise a first input and a second input to said device.

6. The method of claim 5, wherein said first gesture is made with one of a hand and a foot of said user, and said second gesture is made with one of another hand and another foot of said user

7. The method of claim 1, wherein said at least one gesture is compound and includes at least one of (i) a velocity component, (ii) an acceleration component, (iii) a directional component relative to location of said three-dimensional sensor system, and (iv) a gesture comprising movement.

8. The method of claim 1, further including using sound detection to classify at least one said gesture.

9. The method of claim 2, further including segmenting said body part from a remainder of said scene, wherein segmenting includes at least one of (i) using positional information to cluster together pixels representing portions of said body part, (ii) using positional information to cluster together pixels representing portions of said body part based on similar depth information, and (iii) clustering together pixels representing portions of said body part based at least partially on a depth of a discrete region from said system.

10. The method of claim 1, further comprising identifying at least one of (i) a shape of said body part, (ii) a pose orientation of said body part, (iii) a shape of said body part identified by performing statistical analysis on said image, (iv) a pose of said body part, and (iv) a pose and orientation of said body part.

11. The method of claim 2, further including performing a statistical analysis on said image, wherein data analysis includes at least one of (I) identifying a shape of said body part, and (II) performing a statistical analysis on an image portion including said body part segmented from said scene;

wherein performing a statistical analysis includes at least one of (i) creating a histogram using coordinates that identify pixels in said array of pixels, and (ii) creating a histogram using coordinates that identity pixels in said array of pixels on a boundary of said image.

12. The method of claim 1, wherein step (c) includes classifying said gesture based on at least one of (i) shape of said body part, (ii) pose of said body part, (iii) classifying a series of body postures occurring at multiple instances of said time duration interval wherein a combination of said body postures forms said gesture, (iv) using positional information of said body part to recognize said body part is making a gesture to the exclusion of other body parts, (iv) using velocity information of said body part to recognize said body part is making a gesture to the exclusion of other body parts, and (v) using acceleration information of said body part to recognize said body part is making a gesture to the exclusion of other body parts.

13. The method of claim 1, wherein said body part includes at least one of said person'"'"'s hands, and step (c) includes using said positional information to classify a gesture created by at least one of said user'"'"'s hands.

14. The method of claim 1, wherein step (c) includes using positional information to classify a gesture formed by at least one of a finger, a hand, a hand-held object, an arm, a shoulder, an eye ball, an eye lid, a head, a foot, and a leg of said user.

15. The method of claim 1, wherein step (c) classifies said gesture as input for an electronic device selected from a group consisting of a computer, a portable computer, a keyboard, a television system, an audio system, a game console, a mobile phone, a robot, and an appliance.

16. The method of claim 1, wherein said at least one gesture is a dynamic gesture, and detecting at step (b) includes at least one of (i) detecting an occurrence of a first delimiter action signifying when said dynamic gesture starts, (ii) detecting an occurrence of a second delimiter action signifying when said dynamic gesture ends, and (iii) and recognizing that at least one of said first delimiter action and said second delimiter action correspond to a specific posture of a body part of said user.

17. The method of claim 1, wherein at detecting at step (b) includes detecting an occurrence of a delimiter action that creates a designated audible sound.

18. The method of claim 1, further including at least one of (i) communicating to said user gesture classification before step (c) classifies said gesture as said input.

19. The method of claim 17, further including detecting received confirmation from said user that said input is what said user intended to create.

20. The method of claim 18, wherein detecting said received confirmation includes detecting at least one of (i) another gesture by said user, (ii) an audible signal created by said user, and (iii) a manual entry by said user into said device.

21. The method of claim 1, wherein at least one of step (a) and step (c) is provided as instructions on a computer-readable medium, said instructions being executable by at least one processor to perform such step.

3. (canceled)