Dynamic gesture recognition from stereo sequences

• US 20030113018A1
• Filed: 01/23/2003
• Published: 06/19/2003
• Est. Priority Date: 07/18/2001
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Nefian, Ara Victor, Grzesczuk, Radek, Eruhimov, Victor
• Granted Patent: US 7,274,800 B2
• Time in Patent Office: Days
• US Class Current: 382/181
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

According to an embodiment, an apparatus and method are disclosed for dynamic gesture recognition from stereo sequences. In an embodiment, a stereo sequence of images of a subject is obtained and a depth disparity map is generated from the stereo sequence. The system is initiated automatically based upon a statistical model of the upper body of the subject. The upper body of the subject is modeled as three planes, representing the torso and arms of the subject, and three Gaussian components, representing the head and hands of the subject. The system tracks the upper body of the subject using the statistical upper body model and extracts three-dimensional features of the gestures performed. The system recognizes the gestures using recognition units, which, under a particular embodiment, utilizes hidden Markov models for the three-dimensional gestures.

Claims

1. A method comprising:

capturing a sequence of stereo images, the stereo images including at least a. portion of a subject performing a dynamic gesture;

obtaining depth disparities relating to the stereo images;

tracking the subject;

extracting three-dimensional features from the stereo images; and

interpreting the dynamic gesture performed by the subject.

2. The method of claim 1, further comprising segmenting an image of the subject into subparts.

3. The method of claim 2, wherein the subparts represent at least the torso, head, arms, and hands of the subject.

4. The method of claim 1, further comprising automatically initializing parameters of a statistical model of the subject.

5. The method of claim 4, wherein the statistical model of the subject models the arms and torso of the subject as planes.

6. The method of claim 4, wherein the statistical model of the subject models the head and hands of the subject as Gaussian components.

7. The method of claim 1, further comprising removing the background from the stereo images.

8. The method of claim 7, wherein removing the background from the stereo images comprises eliminating any portion of the stereo images that is more than a given distance away from a location.

9. The method of claim 1, wherein the stereo images are captured using a stereo camera.

10. The method of claim 1, wherein obtaining depth disparities comprises generating a depth disparity map.

11. The method of claim 1, wherein interpreting the dynamic gesture comprises comparing the dynamic gesture to a three-dimensional model of a gesture.

12. The method of claim 11, wherein comparing the dynamic gesture to a three-dimensional model of a gesture includes the use of hidden Markov models of three-dimensional gestures.

13. A gesture recognition system comprising:

an imaging device to capture a sequence of three-dimensional images of a least a portion of a subject and a background, the subject performing a dynamic gesture;

a processor to perform operations comprising;

processing a set of depth disparities relating to the stereo images;

tracking the subject;

extracting three-dimensional features from the subject; and

interpreting the dynamic gesture performed by the subject.

14. The gesture recognition system of claim 13, wherein the imaging device is a stereo video camera.

15. The gesture recognition system of claim 13, wherein the processor further performs operations comprising removing the background from the sequence of stereo images.

16. The gesture recognition system of claim 15, wherein removing the background from the sequence of stereo images comprises eliminating any portion of the images that is farther away from the imaging device than a given distance.

17. The gesture recognition system of claim 13, wherein the processor further performs operations comprising segmenting an image of the subject into subparts.

18. The gesture recognition system of claim 17, wherein the subparts represent at least the torso, head, arms, and hands of the subject.

19. The gesture recognition system of claim 13, wherein the processor further performs operations comprising automatically initializing parameters of a statistical model of the subject.

20. The gesture recognition system of claim 19, wherein the statistical model of the subject models the arms and torso of the subject as planes.

21. The gesture recognition system of claim 19, wherein the statistical model of the subject models the head and hands of the subject as Gaussian components.

22. The gesture recognition system of claim 13, wherein interpreting the dynamic gesture performed by the subject comprises comparing the dynamic gesture to a three-dimensional model of a gesture.

23. The gesture recognition system of claim 22, wherein comparing the dynamic gesture to a three-dimensional model of a gesture includes the use of hidden Markov models of three-dimensional gestures.

24. A machine-readable medium having stored thereon data representing sequences of instruction that, when executed by a machine, cause the machine to perform operations comprising:

capturing a sequence of stereo images, the stereo images including at least a portion of a subject performing a dynamic gesture;

obtaining depth disparities relating to the stereo images;

tracking the subject;

extracting three-dimensional features from the stereo images; and

interpreting the dynamic gesture performed by the subject.

25. The medium of claim 24, further comprising sequences of instruction that, when executed by a machine, cause the machine to perform operations comprising segmenting an image of the subject into subparts.

26. The medium of claim 25, wherein the subparts represent at least the torso, head, arms, and hands of the subject.

27. The medium of claim 24, further comprising sequences of instruction that, when executed by a machine, cause the machine to perform operations comprising automatically initializing parameters of a statistical model of the subject.

28. The medium of claim 27, wherein the statistical model of the subject models the arms and torso of the subject as planes.

29. The medium of claim 27, wherein the statistical model of the subject models the head and hands of the subject as Gaussian components.

30. The medium of claim 24, further comprising sequences of instruction that, when executed by a machine, cause the machine to perform operations comprising removing the background from the stereo images.

31. The medium of claim 30, wherein removing the background from the stereo images comprises eliminating any portion of the stereo images that is more than a given distance away from a location.

32. The medium of claim 24, wherein the stereo images are captured using a stereo camera.

33. The medium of claim 24, wherein obtaining depth disparities comprises generating a depth disparity map.

34. The medium of claim 24, wherein interpreting the dynamic gesture comprises comparing the dynamic gesture to a three-dimensional model of a gesture.

35. The medium of claim 34, wherein comparing the dynamic gesture to a three-dimensional model of a gesture includes the use of hidden Markov models of three-dimensional gestures.