Gesture recognition system using depth perceptive sensors

• US 7,340,077 B2
• Filed: 02/18/2003
• Issued: 03/04/2008
• Est. Priority Date: 02/15/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Canesta Incorporated (Microsoft Corporation)
• Inventors: Sürücü, Fahri, Gokturk, Salih Burak, Tomasi, Carlo, Rafii, Abbas
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US10/369,999
• Publication Number:
  US 20030156756A1
• Time in Patent Office: 1,841 Days
• Field of Search: 382/103, 382/106, 382/107, 382/154, 382/162, 382/168, 382/170, 382/171, 382/184, 382/189, 382/194, 382/199, 382/203, 382/209, 382/224, 382/232, 382/256, 382/260, 382/274, 382/285, 382291-298, 382/305, 382/312, 382/132, 382/863, 382/473, 382/208, 715/863, 345/473, 701/208, 348/208.14
• US Class Current: 382/103
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04L 2012/2841 Wireless

Abstract

Three-dimensional position information is used to identify the gesture created by a body part of interest. At one or more instances of an interval, the posture of a body part is recognized, based on the shape of the body part and its position and orientation. The posture of the body part over each of the one or more instances in the interval are recognized as a combined gesture. The gesture is classified for determining an input into a related electronic device.

Claims

1. A method to enable a person to interact with an electronic device by way of a gesture by a body part of said person, the method including the following steps:

(a) obtaining position information for a plurality of discrete regions on said body part, said position information indicating a depth of each discrete region on said body part relative to a reference, said position information obtained at a plurality of instances during a given time duration interval;

(b) detecting from position information obtained at step (a) start time and end time of a dynamic gesture by said body part, wherein said given time duration interval is defined between said start time and said end time; and

(c) classifying said dynamic gesture formed by said body part as an input for interacting with said electronic device using position information obtained during said given time duration interval at each instance in said plurality of instances.

2. The method of claim 1, wherein step (a) includes providing a pixel array to capture an image of a scene including said body part, wherein each pixel in said array corresponds to a discrete region of said scene.

3. The method of claim 2, further including segmenting said body part from a remainder of said scene, wherein segmenting is carried out in a manner selected from a group consisting of (i) using position information to cluster together pixels representing portions of said body part, (ii) using position information to cluster together pixels representing portions of said body part based on similar depth information, and (iii) clustering together pixels representing portions of said body part based at least partially on a depth of a discrete region from said reference.

4. The method of claim 2, further including performing a statistical analysis on said image in a manner selected from a group consisting of (I) identifying a shape of said body part, and (II) performing a statistical analysis on an image portion including said body part segmented from said scene;

wherein performing a statistical analysis includes at least one of (i) creating a histogram using coordinates that identify pixels in said pixel array, and (ii) creating a histogram using coordinates that identity pixels in said pixel array on a boundary of said image.

5. The method of claim 1, further including segmenting said body part from a remainder of a scene including said body part.

6. The method of claim 1, further comprising identifying at least one of (i) a shape of said body part, (ii) a shape of said body part identified by performing statistical analysis on said image, (iii) a pose of said body part, and (iv) a pose and orientation of said body part.

7. The method of claim 1, wherein step (c) includes classifying said gesture based on at least one of (i) shape of said body part, (ii) pose of said body part, and (iii) classifying a series of body postures occurring at multiple instances of said time duration interval wherein a combination of said body postures forms said gesture.

8. The method of claim 1, wherein said body part includes at least one of said person'"'"'s hands, and step (c) includes using said position information to classify a gesture created by at least one of said person'"'"'s hands.

9. The method of claim 1, wherein step (c) includes using position information to classify a gesture formed by at least one of a finger, a hand, an arm, a shoulder, an eye ball, an eye lid, a head, a foot, and a leg of said person.

10. The method of claim 1, wherein step (c) classifies said gesture as input for an electronic device selected from a group consisting of a portable computer, a television system, an audio system, a game console, a mobile phone, a robot, and an appliance.

11. The method of claim 1, wherein step (a) includes obtaining position information at a plurality of instances during a given said time duration interval.

12. The method of claim 11, wherein step (c) includes using position information obtained at each instance to classify a gesture formed by movement of said body part during said time duration.

13. The method of claim 1, wherein at step (b), detecting start of a gesture includes detecting an occurrence of a first delimiter action signifying when a gesture starts.

14. The method of claim 1, wherein at step (b), detecting when a gesture ends includes detecting an occurrence of a second delimiter action signifying when a gesture ends.

15. The method of claim 1, wherein at step (b), detecting at least one of start and end of a gesture includes detecting an occurrence of a delimiter action that creates a designated audible sound.

16. The method of claim 15, wherein at least one of said first delimiter action and said second delimiter action corresponds to formation of a specific posture of one of the said body part of said person and another body part of said person.

17. The method of claim 1, further including indicating gesture classification to said person before classifying said gesture as said input.

18. The method of claim 17, further including receiving confirmation from said person that said input is what said person intended to enter.

19. The method of claim 18, further including detecting said confirmation.

20. The method of claim 19, wherein detecting said confirmation includes detecting at least one of another gesture by said person, an audible signal created by said person, and a manual entry by said person into said device.

21. The method of claim 1, wherein at least one of step (a) and step (c) is provided as instructions on a computer-readable medium, said instructions being executable by at least one processor to perform such step.