METHOD AND SYSTEM FOR GESTURE CLASSIFICATION

• US 20090085864A1
• Filed: 10/02/2007
• Published: 04/02/2009
• Est. Priority Date: 10/02/2007
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Omek Interactive Ltd. (Intel Corporation)
• Inventors: Bleiweiss, Amit, Kutliroff, Gershom
• Granted Patent: US 7,970,176 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The present invention is a method of identifying a user'"'"'s gestures for use in an interactive game application. Videocamera images of the user are obtained, and feature point locations of a user'"'"'s body are identified in the images. A similarity measure is used to compare the feature point locations in the images with a library of gestures. The gesture in the library corresponding to the largest calculated similarity measure which is greater than a threshold value of the gesture is identified as the user'"'"'s gesture. The identified gesture may be integrated into the user'"'"'s movements within a virtual gaming environment, and visual feedback is provided to the user.

Claims

1. A method of classifying gestures comprising:

acquiring images of a subject making movements; and

using a similarity measure and a database of catalogued gestures to identify the movements as a known gesture.

2. The method as claimed in claim 1, wherein the similarity measure quantifies a similarity between the movements and one of the catalogued gestures.

3. A method of classifying gestures comprising:

acquiring a first plurality of images of a subject;

identifying a first set of feature points in the first plurality of images;

calculating one or more similarity measures, wherein the one or more similarity measures quantify a similarity of the first set of feature points in the first plurality of images to one or more second sets of feature points of one or more catalogued gestures captured in one or more second plurality of images; and

using one or more threshold values for the one or more similarity measures associated with the one or more catalogued gestures to determine if the subject made the one or more catalogued gestures.

4. The method as claimed in claim 3 further comprising selecting an identified gesture as the one or more catalogued gestures resulting in a largest similarity measure.

5. The method as claimed in claim 3, wherein the first plurality of images comprises a plurality of depth images and a plurality of color images.

6. The method as claimed in claim 5, wherein each of the plurality of color images comprises three channels, and each channel corresponds to a different color.

7. The method as claimed in claim 3, wherein the first set of feature points are points on the subject'"'"'s body such as joints.

8. The method as claimed in claim 3, wherein the one or more second plurality of images is created by recording multiple series of exemplary images of one or more users performing the one or more catalogued gesture multiple times and averaging over the multiple series of exemplary images for each of the one or more catalogued gestures.

9. The method as claimed in claim 8, wherein a sensor is placed on each feature point on the one or more users while recording the multiple series of exemplary images, and the multiple series of exemplary images are post-processed before averaging over the multiple series of exemplary images for each of the one or more catalogued gestures.

10. The method as claimed in claim 8, wherein each exemplary image includes a depth image and a color image, and a three-dimensional coordinate is calculated for each feature point and smoothed temporally and spatially before averaging over the multiple series of exemplary images for each of the one or more catalogued gestures.

11. The method as claimed in claim 3, wherein the similarity measure is calculated using a feature-point-dependent weighting function and a time-dependent weighting function.

12. A system for classifying gestures comprising:

a sensor for acquiring images of a subject making movements;

a database of catalogued gestures; and

a processor for using a similarity measure and the database to identify the movements as a known gesture.

13. The system as claimed in claim 12, wherein the similarity measure quantifies a similarity between the movements and one of the catalogued gestures.

14. A system for classifying gestures comprising:

a sensor for acquiring a first plurality of images of a subject;

a database of known gestures; and

a processor for identifying a first set of feature points in the first plurality of images;

calculating a similarity measure, wherein the similarity measure quantifies a similarity of the first set of feature points in the first plurality of images to a second set of feature points of a known gesture captured in a second plurality of images; and

using a threshold value of the similarity measure associated with the known gesture to determine if the subject made the known gesture.

15. The system as claimed in claim 14, wherein the first plurality of images comprises a plurality of depth images and a plurality of color images.

16. The system as claimed in claim 15, wherein each of the plurality of color images comprises three channels, and each channel corresponds to a different color.

17. The system as claimed in claim 14, wherein the second plurality of images is created by recording multiple series of exemplary images of one or more users performing the known gesture multiple times and averaging over the multiple series of exemplary images.

18. The system as claimed in claim 17, wherein a sensor is placed on each feature point on the one or more users while recording the multiple series of exemplary images, and the multiple series of exemplary images are post-processed before averaging over the multiple series of exemplary images.

19. The system as claimed in claim 17, wherein each exemplary image includes a depth image and a color image, and a three-dimensional coordinate is calculated for each feature point and smoothed temporally and spatially before averaging over the multiple series of exemplary images.

20. The system as claimed in claim 14, wherein the similarity measure is calculated using a feature-point-dependent weighting function and a time-dependent weighting function.

21. A system for classifying gestures comprising:

means for acquiring images of a subject making movements; and

means for using a similarity measure and a database of catalogued gestures to identify the movements as a known gesture.

22. The system as claimed in claim 21, wherein the similarity measure quantifies a similarity between the movements and one of the catalogued gestures.

23. A method of allowing a player to interact with a virtual environment comprising:

acquiring images of the player making movements;

using a similarity measure and a database of catalogued gestures to identify the movements as a known gesture; and

integrating and displaying a virtual image of the player making the known gesture within the virtual environment.

24. The method as claimed in claim 23 wherein the virtual environment simulates a game.

25. The method as claimed in claim 23 wherein the virtual environment simulates a sport.

26. The method as claimed in claim 23 wherein the virtual environment simulates a fitness program.

27. A method of allowing a player to interact with a virtual environment comprising:

acquiring a first plurality of images of the player;

identifying a first set of feature points in the first plurality of images;

calculating one or more similarity measures, wherein the one or more similarity measures quantify a similarity of the first set of feature points in the first plurality of images to one or more second sets of feature points of one or more catalogued gestures captured in one or more second plurality of images;

using one or more threshold values for the one or more similarity measures associated with the one or more catalogued gestures to determine if the player made the one or more catalogued gestures; and

selecting an identified gesture as the one or more catalogued gestures resulting in a largest similarity measure;

integrating and displaying a virtual image of the player making the identified gesture within the virtual environment.

28. The method as claimed in claim 27 wherein the virtual environment simulates a game.

29. The method as claimed in claim 27 wherein the virtual environment simulates a sport.

30. The method as claimed in claim 27 wherein the virtual environment simulates a fitness program.

31. A system for allowing a player to interact with a virtual environment comprising:

a sensor for acquiring images of the player making movements;

a database of catalogued gestures;

a processor for using a similarity measure and the database to identify the movements as a known gesture; and

a display for displaying a virtual image of the player making the known gesture within the virtual environment.

32. The system as claimed in claim 31 wherein the virtual environment simulates a game.

33. The system as claimed in claim 31 wherein the virtual environment simulates a sport.

34. The system as claimed in claim 31 wherein the virtual environment simulates a fitness program.

35. A method of allowing a first player to interact with a second player in a virtual environment comprising:

acquiring a first set images of the first player making a first set of movements and a second set images of the second player making a second set of movements;

using a similarity measure and a database of catalogued gestures to identify the first set of movements as a first known gesture and the second set of movements as a second known gesture; and

integrating and displaying a virtual image of the first player making the first known gesture and the second player making the second known gesture within the virtual environment.

36. The method as claimed in claim 35 wherein the virtual environment simulates a game.

37. The method as claimed in claim 35 wherein the virtual environment simulates a sport.

38. The method as claimed in claim 35 wherein the virtual environment simulates a fitness program.

39. A system for allowing a first player to interact with a second player in a virtual environment comprising:

a first sensor for acquiring a first set of images of the first player making a first set of movements;

a second sensor for acquiring a second set of images of the second player making a second set of movements;

a database of catalogued gestures;

a processor for using a similarity measure and the database to identify the first set of movements as a first known gesture and the second set of movements as a second known gesture;

a first display and a second display, each displaying a virtual image of the first player making the first known gesture and the second player making the second known gesture within the virtual environment; and

a network to provide a communications between the first sensor, the second sensor, the database, the processor, the first display, and the second display.

40. The system as claimed in claim 39 wherein the virtual environment simulates a game.

41. The system as claimed in claim 39 wherein the virtual environment simulates a sport.

42. The system as claimed in claim 39 wherein the virtual environment simulates a fitness program.