Dynamic gesture recognition from stereo sequences

• US 7,274,800 B2
• Filed: 01/23/2003
• Issued: 09/25/2007
• Est. Priority Date: 07/18/2001
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Eruhimov, Victor, Nefian, Ara Victor, Grzesczuk, Radek
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US10/349,872
• Publication Number:
  US 20030113018A1
• Time in Patent Office: 1,706 Days
• Field of Search: 382/100, 382/103, 382/107, 382/154, 715/863, 348/169, 348/208.1
• US Class Current: 382/103
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

According to an embodiment, an apparatus and method are disclosed for dynamic gesture recognition from stereo sequences. In an embodiment, a stereo sequence of images of a subject is obtained and a depth disparity map is generated from the stereo sequence. The system is initiated automatically based upon a statistical model of the upper body of the subject. The upper body of the subject is modeled as three planes, representing the torso and arms of the subject, and three Gaussian components, representing the head and hands of the subject. The system tracks the upper body of the subject using the statistical upper body model and extracts three-dimensional features of the gestures performed. The system recognizes the gestures using recognition units, which, under a particular embodiment, utilizes hidden Markov models for the three-dimensional gestures.

Claims

1. A method comprising:

capturing a sequence of stereo images, the stereo images including at least a portion of a subject performing a dynamic gesture;

obtaining depth disparities relating to the stereo images;

automatically initializing parameters of a statistical model of the subject based upon matching an image of the subject to the statistical model;

tracking the subject using the statistical model of the subject;

extracting three-dimensional features from the stereo images; and

interpreting the dynamic gesture performed by the subject.

2. The method of claim 1, further comprising segmenting an image of the subject into subparts.

3. The method of claim 2, wherein the subparts represent at least the torso, head, arms, and hands of the subject.

4. The method of claim 1, wherein the statistical model of the subject models the arms and torso of the subject as planes.

5. The method of claim 1, wherein the statistical model of the subject models the head and hands of the subject as Gaussian components.

6. The method of claim 1, further comprising removing the background from the stereo images.

7. The method of claim 6, wherein removing the background from the stereo images comprises eliminating any portion of the stereo images that is more than a given distance away from a location.

8. The method of claim 1, wherein the stereo images are captured using a stereo camera.

9. The method of claim 1, wherein obtaining depth disparities comprises generating a depth disparity map.

10. The method of claim 1, wherein interpreting the dynamic gesture comprises comparing the dynamic gesture to a three-dimensional model of a gesture.

11. The method of claim 10, wherein comparing the dynamic gesture to a three-dimensional model of a gesture includes the use of hidden Markov models of three-dimensional gestures.

12. A gesture recognition system comprising:

an imaging device to capture a sequence of three-dimensional images of a least a portion of a subject and a background, the subject performing a dynamic gesture;

a processor to perform operations comprising;

processing a set of depth disparities relating to the stereo images;

automatically initializing parameters of a statistical model of the subject based upon matching an image of the subject to the statistical model;

tracking the subject using the statistical model of the subject;

extracting three-dimensional features from the subject; and

interpreting the dynamic gesture performed by the subject.

13. The gesture recognition system of claim 12, wherein the imaging device is a stereo video camera.

14. The gesture recognition system of claim 12, wherein the processor further performs operations comprising removing the background from the sequence of stereo images.

15. The gesture recognition system of claim 14, wherein removing the background from the sequence of stereo images comprises eliminating any portion of the images that is farther away from the imaging device than a given distance.

16. The gesture recognition system of claim 12, wherein the processor further performs operations comprising segmenting an image of the subject into subparts.

17. The gesture recognition system of claim 16, wherein the subparts represent at least the torso, head, arms, and hands of the subject.

18. The gesture recognition system of claim 12, wherein the statistical model of the subject models the arms and torso of the subject as planes.

19. The gesture recognition system of claim 12, wherein the statistical model of the subject models the head and hands of the subject as Gaussian components.

20. The gesture recognition system of claim 12, wherein interpreting the dynamic gesture performed by the subject comprises comparing the dynamic gesture to a three-dimensional model of a gesture.

21. The gesture recognition system of claim 20, wherein comparing the dynamic gesture to a three-dimensional model of a gesture includes the use of hidden Markov models of three-dimensional gestures.

22. A machine-readable medium having stored thereon data representing sequences of instruction that, when executed by a machine, cause the machine to perform operations comprising:

capturing a sequence of stereo images, the stereo images including at least a portion of a subject performing a dynamic gesture;

obtaining depth disparities relating to the stereo images;

automatically initializing parameters of a statistical model of the subject based upon matching an image of the subject to the statistical model;

tracking the subject using the statistical model of the subject;

extracting three-dimensional features from the stereo images; and

interpreting the dynamic gesture performed by the subject.

23. The medium of claim 22, further comprising sequences of instruction that, when executed by a machine, cause the machine to perform operations comprising segmenting an image of the subject into subparts.

24. The medium of claim 23, wherein the subparts represent at least the torso, head, arms, and hands of the subject.

25. The medium of claim 22, wherein the statistical model of the subject models the arms and torso of the subject as planes.

26. The medium of claim 22, wherein the statistical model of the subject models the head and hands of the subject as Gaussian components.

27. The medium of claim 22, further comprising sequences of instruction that, when executed by a machine, cause the machine to perform operations comprising removing the background from the stereo images.

28. The medium of claim 27, wherein removing the background from the stereo images comprises eliminating any portion of the stereo images that is more than a given distance away from a location.

29. The medium of claim 22, wherein the stereo images are captured using a stereo camera.

30. The medium of claim 22, wherein obtaining depth disparities comprises generating a depth disparity map.

31. The medium of claim 22, wherein interpreting the dynamic gesture comprises comparing the dynamic gesture to a three-dimensional model of a gesture.

32. The medium of claim 31, wherein comparing the dynamic gesture to a three-dimensional model of a gesture includes the use of hidden Markov models of three-dimensional gestures.