SYSTEM AND METHOD FOR GESTURE RECOGNITION

• US 20130294651A1
• Filed: 12/29/2010
• Published: 11/07/2013
• Est. Priority Date: 12/29/2010
• Status: Active Grant

• Current Assignee: Thomson Licensing (Vantiva SA)
• Original Assignee: Thomson Licensing (Vantiva SA)
• Inventors: Zhou, Jie, Cheng, Pu
• Granted Patent: US 9,323,337 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/85 Markov-related models; Markov random fields
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system and method for gesture spotting and recognition are provided. Systems and methods are also provided employing Hidden Markov Models (HMM) and geometrical feature distributions of a hand trajectory of a user to achieve adaptive gesture recognition. The system and method provide for acquiring a sequence of input images of a specific user and recognizing a gesture of the specific user from the sequence of input images based on a gesture model and geometrical features extracted from a hand trajectory of the user. State transition points of the gesture model are detected and the geometrical features of the hand trajectory of the user are extracted based on the relative positions of the detected state transition points and a starting point of the gesture. The system and method further provide for adapting the gesture model and geometrical feature distribution for the specific user based on adaptation data.

Claims

1. A method for recognizing a gesture performed by a user, the method comprising:

acquiring a sequence of input images of a specific user; and

recognizing a gesture of the specific user from the sequence of input images based on a gesture model and geometrical features extracted from a hand trajectory of the user.

2. The method as in claim 1, wherein the gesture model is a Hidden Markov Model (HMM).

3. The method as in claim 1, wherein the recognizing step further comprises:

acquiring a sequence of input images of a user;

detecting abrupt changes of motion parameters in the sequence of input images as candidate starting points;

matching the sequence of input images to a gesture model;

detecting state transition points of the matched gesture model;

extracting geometrical features of a hand trajectory of the user based on the detected state transition points and candidate starting points;

determining an endpoint and gesture class of the gesture based on matched gesture model and the extracted geometrical features; and

backtracking through the images from the determined endpoint of the gesture to determine an actual starting point of the gesture.

4. The method as in claim 3, wherein the determining the endpoint of the gesture further comprises:

determining a gesture model matching score for each frame of the sequence of input images; and

selecting the frame having the gesture model matching score greater than a first predetermined threshold as the frame containing the endpoint of the gesture.

5. The method as in claim 4, further comprising pruning the gesture model matching score of at least one frame of the sequence of input images based on the extracted geometrical features and determined gesture class.

6. The method as in claim 4, further comprising selecting a second gesture model for matching if a probability of the extracted geometrical feature appearing in a frame is below a second predetermined threshold.

7. The method as in claim 6, wherein the second predetermined threshold is based on the determined gesture class.

8. The method as in claim 3, wherein the backtracking through the images is performed by a Dynamic Programming (DP) function.

9. The method as in claim 1, wherein the recognizing step further comprises:

detecting and tracking at least one object in each frame of the sequence of input images; and

matching features of the detected and tracked at least one object to determine the gesture model.

10. The method as in claim 9, wherein the recognizing step further comprises:

detecting state transition points of the gesture model; and

extracting the geometrical features of the hand trajectory of the user based on the relative positions of the detected state transition points and a starting point of the gesture.

11. The method as in claim 10, wherein the recognizing step further comprises:

computing a match score for the detected and tracked at least one object for each gesture class using the gesture model and extracted geometrical features; and

classifying the detected and tracked at least one object as the gesture class with the highest match score.

12. The method as in claim 1, further comprising adapting the gesture model and geometrical feature distribution for the specific user based on gestures performed by the specific user.

13. The method as in claim 12, wherein a maximum likelihood linear regression (MLLR) function is employed to update the gesture model with gesture samples from the specific user.

14. The method as in claim 13, further comprising incrementally learning the geometrical feature distribution for the specific user by re-estimating a mean and covariance matrix of the geometrical feature distribution over a predetermined number of adaptation samples.

15. The method as in claim 14, further comprising storing the updated gesture model and learned geometrical feature distribution for the specific user.

16. A system for recognizing a gesture performed by a user comprising:

an image capture device that acquires a sequence of input images of a specific user; and

a gesture recognizer that recognizes a gesture of the specific user from the sequence of input images based on a gesture model and geometrical features extracted from a hand trajectory of the user.

17. The system as in claim 16, wherein the gesture model is a Hidden Markov Model (HMM).

18. A system as in claim 16, further comprising:

an object detector and tracker that detects abrupt changes of motion parameters in the sequence of input images as candidate starting points;

a model matcher that matches the sequence of input images to a gesture model;

a transaction detector that detects state transition points of the matched gesture model; and

a feature extractor that extracts geometrical features of a hand trajectory of the user based on the detected state transition points and candidate starting points;

wherein

the gesture recognizer determines an endpoint and gesture class of the gesture based on matched gesture model and the extracted geometrical features and backtracks through the images from the determined endpoint of the gesture to determine an actual starting point of the gesture.

19. The system as in claim 18, wherein the gesture recognizer is further configured for determining a gesture model matching score for each frame of the sequence of input images and selecting the frame having the gesture model matching score greater than a first predetermined threshold as the frame containing the endpoint of the gesture.

20. The system as in claim 19, further comprising a pruner that rejects the gesture model matching score of at least one frame of the sequence of input images based on the extracted geometrical features and determined gesture class.

21. The system as in claim 19, wherein the model matcher is further configured for selecting a second gesture model for matching if a probability of the extracted geometrical feature appearing in a frame is below a second predetermined threshold.

22. The system as in claim 21, wherein the second predetermined threshold is based on the determined gesture class.

23. The system as in claim 18, wherein gesture recognizer includes a Dynamic Programming (DP) function for backtracking through the images.

24. The system as in claim 16, further comprising:

an object detector and tracker that detects and tracks at least one object in each frame of the sequence of input images; and

a model matcher that match features of the detected and tracked at least one object to determine the gesture model.

25. The system as in claim 24, further comprising:

a transition detector that detects state transition points of the gesture model; and

a feature extractor that extracts the geometrical features of the hand trajectory of the user based on the relative positions of the detected state transition points and a starting point of the gesture.

26. The system as in claim 25, wherein the gesture recognizer is further configured for computing a match score for the detected and tracked at least one object for each gesture class using the gesture model and extracted geometrical features and classifying the detected and tracked at least one object as the gesture class with the highest match score.

27. The system as in claim 16, wherein the gesture recognizer is further configured for adapting the gesture model and geometrical feature distribution for the specific user based on gestures performed by the specific user.

28. The system as in claim 27, wherein the gesture recognizer includes a maximum likelihood linear regression (MLLR) function to update the gesture model with gesture samples from the specific user.

29. The system as in claim 28, wherein the gesture recognizer is further configured for incrementally learning the geometrical feature distribution for the specific user by re-estimating a mean and covariance matrix of the geometrical feature distribution over a predetermined number of adaptation samples.

30. The system as in claim 29, wherein the gesture recognizer is further configured for storing the updated gesture model and learned geometrical feature distribution for the specific user in storage device.