Method and system for gesture classification

• US 7,970,176 B2
• Filed: 10/02/2007
• Issued: 06/28/2011
• Est. Priority Date: 10/02/2007
• Status: Expired due to Fees

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Omek Interactive, Inc.
• Inventors: Kutliroff, Gershom, Bleiweiss, Amit
• Primary Examiner(s): AZARIAN, SEYED H
• Application Number:
  US11/866,280
• Publication Number:
  US 20090085864A1
• Time in Patent Office: 1,365 Days
• Field of Search: 382/100, 382/103, 382/107, 382/155, 382/162, 382/168, 382/173, 382/181, 382/189, 382/199, 382/203, 382/209, 382/224, 382/232, 382/254, 382/274, 382/276, 382287-291, 382/305, 382/312, 345/853, 359/630, 482/8, 709/206, 715/863
• US Class Current: 382/103
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The present invention is a method of identifying a user'"'"'s gestures for use in an interactive game application. Videocamera images of the user are obtained, and feature point locations of a user'"'"'s body are identified in the images. A similarity measure is used to compare the feature point locations in the images with a library of gestures. The gesture in the library corresponding to the largest calculated similarity measure which is greater than a threshold value of the gesture is identified as the user'"'"'s gesture. The identified gesture may be integrated into the user'"'"'s movements within a virtual gaming environment, and visual feedback is provided to the user.

Claims

1. A method of classifying gestures comprising:

acquiring a first plurality of images of a subject;

identifying a first set of feature points in the first plurality of images;

calculating one or more similarity measures, wherein the one or more similarity measures quantify a similarity of the first set of feature points in the first plurality of images to one or more second sets of feature points of one or more catalogued gestures captured in one or more second plurality of images;

using one or more threshold values for the one or more similarity measures associated with the one or more catalogued gestures to determine if the subject made the one or more catalogued gestures; and

selecting an identified gesture as the one or more catalogued gestures resulting in a largest similarity measure.

2. The method as claimed in claim 1, wherein the first plurality of images comprises a plurality of depth images and a plurality of color images.

3. The method as claimed in claim 2, wherein each of the plurality of color images comprises three channels, and each channel corresponds to a different color.

4. The method as claimed in claim 1, wherein the first set of feature points are points on the subject'"'"'s body such as joints.

5. The method as claimed in claim 1, wherein the one or more second plurality of images is created by recording multiple series of exemplary images of one or more users performing the one or more catalogued gesture multiple times and averaging over the multiple series of exemplary images for each of the one or more catalogued gestures.

6. The method as claimed in claim 5, wherein a sensor is placed on each feature point on the one or more users while recording the multiple series of exemplary images, and the multiple series of exemplary images are post-processed before averaging over the multiple series of exemplary images for each of the one or more catalogued gestures.

7. The method as claimed in claim 5, wherein each exemplary image includes a depth image and a color image, and a three-dimensional coordinate is calculated for each feature point and smoothed temporally and spatially before averaging over the multiple series of exemplary images for each of the one or more catalogued gestures.

8. The method as claimed in claim 1, wherein the similarity measure is calculated using a feature-point-dependent weighting function and a time-dependent weighting function.

9. A system for classifying gestures comprising:

a sensor for acquiring a first plurality of images of a subject;

a database of known gestures; and

a processor for identifying a first set of feature points in the first plurality of images;

calculating a similarity measure, wherein the similarity measure quantifies a similarity of the first set of feature points in the first plurality of images to a second set of feature points of a known gesture captured in a second plurality of images, wherein the second plurality of images is created by recording multiple series of exemplary images of one or more users performing the known gesture multiple times and averaging over the multiple series of exemplary images; and

using a threshold value of the similarity measure associated with the known gesture to determine if the subject made the known gesture.

10. The system as claimed in claim 9, wherein the first plurality of images comprises a plurality of depth images and a plurality of color images.

11. The system as claimed in claim 10, wherein each of the plurality of color images comprises three channels, and each channel corresponds to a different color.

12. The system as claimed in claim 9, wherein a sensor is placed on each feature point on the one or more users while recording the multiple series of exemplary images, and the multiple series of exemplary images are post-processed before averaging over the multiple series of exemplary images.

13. The system as claimed in claim 9, wherein each exemplary image includes a depth image and a color image, and a three-dimensional coordinate is calculated for each feature point and smoothed temporally and spatially before averaging over the multiple series of exemplary images.

14. The system as claimed in claim 9, wherein the similarity measure is calculated using a feature-point-dependent weighting function and a time-dependent weighting function.

15. A method of allowing a player to interact with a virtual environment comprising:

acquiring a first plurality of images of the player;

identifying a first set of feature points in the first plurality of images;

calculating one or more similarity measures, wherein the one or more similarity measures quantify a similarity of the first set of feature points in the first plurality of images to one or more second sets of feature points of one or more catalogued gestures captured in one or more second plurality of images;

using one or more threshold values for the one or more similarity measures associated with the one or more catalogued gestures to determine if the player made the one or more catalogued gestures; and

selecting an identified gesture as the one or more catalogued gestures resulting in a largest similarity measure;

integrating and displaying a virtual image of the player making the identified gesture within the virtual environment.

16. The method as claimed in claim 15 wherein the virtual environment simulates a game.

17. The method as claimed in claim 15 wherein the virtual environment simulates a sport.

18. The method as claimed in claim 15 wherein the virtual environment simulates a fitness program.

19. A system for allowing a first player to interact with a second player in a virtual environment comprising:

a first sensor for acquiring a first set of images of the first player making a first set of movements;

a second sensor for acquiring a second set of images of the second player making a second set of movements;

a database of catalogued gestures;

a processor for using a similarity measure and the database to identify the first set of movements as a first known gesture and the second set of movements as a second known gesture;

a first display and a second display, each displaying a virtual image of the first player making the first known gesture and the second player making the second known gesture within the virtual environment; and

a network to provide a communications between the first sensor, the second sensor, the database, the processor, the first display, and the second display.

20. The system as claimed in claim 19 wherein the virtual environment simulates a game.

21. The system as claimed in claim 19 wherein the virtual environment simulates a sport.

22. The system as claimed in claim 19 wherein the virtual environment simulates a fitness program.

23. A computer memory storing gesture classifying instructions for execution by a computer processor, wherein the gesture classifying instructions comprise:

acquiring a first plurality of images of a subject;

identifying a first set of feature points in the first plurality of images;

calculating one or more similarity measures, wherein the one or more similarity measures quantify a similarity of the first set of feature points in the first plurality of images to one or more second sets of feature points of one or more catalogued gestures captured in one or more second plurality of images;

using one or more threshold values for the one or more similarity measures associated with the one or more catalogued gestures to determine if the subject made the one or more catalogued gestures; and

selecting an identified gesture as the one or more catalogued gestures resulting in a largest similarity measure.

24. The computer memory as claimed in claim 23, wherein the first plurality of images comprises a plurality of depth images and a plurality of color images.

25. The computer memory as claimed in claim 24, wherein a sensor is placed on each feature point on the one or more users while recording the multiple series of exemplary images, and the multiple series of exemplary images are post-processed before averaging over the multiple series of exemplary images for each of the one or more catalogued gestures.

26. The computer memory as claimed in claim 25, wherein each exemplary image includes a depth image and a color image, and a three-dimensional coordinate is calculated for each feature point and smoothed temporally and spatially before averaging over the multiple series of exemplary images for each of the one or more catalogued gestures.

27. The computer memory as claimed in claim 23, wherein each of the plurality of color images comprises three channels, and each channel corresponds to a different color.

28. The computer memory as claimed in claim 23, wherein the first set of feature points are points on the subject'"'"'s body such as joints.

29. The computer memory as claimed in claim 23, wherein the one or more second plurality of images is created by recording multiple series of exemplary images of one or more users performing the one or more catalogued gesture multiple times and averaging over the multiple series of exemplary images for each of the one or more catalogued gestures.

30. The method as claimed in claim 23, wherein the similarity measure is calculated using a feature-point-dependent weighting function and a time-dependent weighting function.