Methods and apparatus for gesture recognition based on templates

• US 6,075,895 A
• Filed: 06/20/1997
• Issued: 06/13/2000
• Est. Priority Date: 06/20/1997
• Status: Expired due to Fees

• Current Assignee: Holoplex Incorporated
• Original Assignee: Holoplex Incorporated
• Inventors: Zhou, Gan, Qiao, Yong, Mok, Fai
• Primary Examiner(s): Bella, Matthew C.
• Application Number:
  US08/879,854
• Time in Patent Office: 1,089 Days
• Field of Search: 382/100, 382/103, 382/155, 382/156, 382/159, 382/195, 382/201, 382/202, 382/203, 382/218, 382/232, 382/236, 382/270, 382/271, 382/272, 382/273, 382/277, 382/278, 382/282 , 382/287, 382/309, 345/358, 345/434, 345/118, 348/40, 348/47, 348/155, 348/157
• US Class Current: 382/218
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

An apparatus and a method for recognizing the gesture of the image of a player by identifying a gesture from a set of specific pre-defined gestures. In one embodiment, the method includes the steps of retrieving a background image that does not include the player'"'"'s image, and a current image that includes the player'"'"'s image and at least a portion of the background image. Then the method removes at least a portion of the background image from the current image to generate the player'"'"'s image, which is mapped directly to a number of templates to generate a number of template outputs. Finally, the method analyzes the template outputs to identify the specific pre-defined gesture that corresponds to the gesture in the image.

Claims

1. A method for recognizing the gesture of the image of a being by identifying a gesture from a set of specific pre-defined gestures, the method comprising the steps of:

retrieving a background image that does not include the being'"'"'s image;

retrieving a current image that includes the being'"'"'s image and at least a portion of the background image;

removing at least a portion of the background image from the current image to generate the being'"'"'s image, which has a plurality of pixels;

mapping the being'"'"'s image directly to a plurality of templates, each having a plurality of pixels, based on matching at least one pixel of the being'"'"'s image upon its corresponding pixel in each of the templates, to generate a plurality of template outputs;

analyzing the template outputs to identify the specific pre-defined gesture that corresponds to the gesture in the image; and

repeating from the step of retrieving to recognize another gesture of the image of the being based on the same background image.

2. A method as recited in claim 1 wherein:

the being is a player; and

the gesture is used in an electronic game.

3. A method as recited in claim 2 wherein the player'"'"'s sound is captured and is re-broadcasted in the electronic game.

4. A method as recited in claim 2 wherein the player'"'"'s face is digitized and is used in the electronic game.

5. A method as recited in claim 1 wherein the background image and the current image are captured by more than one detector, with the different detectors capturing the images from different directions.

6. A method as recited in claim 1 further comprising the steps of:

retrieving a threshold value; and

retrieving a upper limit value; and

wherein;

each image has a plurality of pixels, with each pixel having a value;

the step of removing at least a portion of the background image further comprising the steps of;

comparing the difference between the value of each pixel in the background image and the value of the corresponding pixel in the current image with the threshold value to generate an energy level;

changing the threshold value if the energy level is higher than the upper limit value;

repeating from the step of comparing if the energy level is higher than the upper limit value; and

generating the being'"'"'s image based on the background image, the current image and the threshold value.

7. A method as recited in claim 6 wherein:

the templates are separated into a plurality of sets of templates;

each template has a bar of pixels whose values are non-zero;

the bars in the templates within each set are substantially parallel;

the orientations of the bars in different sets are different; and

combining the bars within each set substantially covers the image.

8. A method as recited in claim 7 wherein:

the set of specific pre-defined gestures can be classified under one or more general pre-defined gestures; and

the step of analyzing the template outputs further includes the steps of;

identifying at least one general pre-defined gesture; and

identifying at least one specific pre-defined gesture in the identified general pre-defined gesture such that the specific pre-defined gesture corresponds to the gesture in the image.

9. A method as recited in claim 8 wherein the step of analyzing the template outputs further includes the step of selecting at least one approach out of the neural network approach and the rule-based approach to identify the specific pre-defined gesture that corresponds to the gesture in the image.

10. A method as recited in claim 9 wherein:

each template is represented by a hologram; and

the step of mapping further comprises the step of optically correlating the being'"'"'s image with the plurality of holograms to generate the template outputs.

11. A method as recited in claim 10 wherein:

the step of mapping further comprises the steps of identifying one specific position on the being'"'"'s image based on the template outputs;

the repeating step identifies the change in the specific position of the being'"'"'s image; and

the step of analyzing the template outputs further comprises the step of analyzing the template outputs based on the change in the specific position of the being'"'"'s image to identify the specific pre-defined gesture that corresponds to the gesture of the image.

12. A method as recited in claim 9 wherein the step of mapping further comprises the steps of:

digitally compressing the being'"'"'s image to a compressed being'"'"'s image;

digitally compressing the templates to a plurality of compressed templates; and

generating the template outputs by a process that includes the step of performing inner products of the compressed being'"'"'s image with the compressed templates.

13. A method for recognizing the gesture of the image of a being by identifying a gesture from a set of specific pre-defined gestures, with the image having a plurality of pixels, and with every pixel having a value, the method comprising the steps of:

retrieving the image of the being generated based on a background image, with the image of the being having a plurality of pixels;

retrieving a plurality of templates wherein;

the templates are separated into a plurality of sets of templates;

each template has a bar of pixels whose values are non-zero;

the bars in the templates within each set are substantially parallel;

the orientations of the bars in different sets are different; and

combining the bars within each set substantially covers the image;

mapping the being'"'"'s image to the plurality of templates based on matching at least one pixel of the being'"'"'s image upon its corresponding pixel in each of the templates, to generate a plurality of template outputs;

analyzing the template outputs to identify the specific pre-defined gesture that corresponds to the gesture in the image; and

repeating from the step of retrieving the image to recognize another gesture based on the same background image.

14. A method as recited in claim 13 wherein before the step of retrieving the image of the being, the method further comprises the steps of:

retrieving the background image that does not include the being'"'"'s image;

retrieving a current image that includes the being'"'"'s image and at least a portion of the background image; and

removing at least a portion of the background image from the current image to generate the being'"'"'s image.

15. A method for recognizing the gesture of the image of a being by identifying a gesture from a set of specific pre-defined gestures, which are classified into one or more general pre-defined gestures, the method comprising the steps of:

retrieving the image of the being generated based on a background image, with the image having a plurality of pixels;

mapping the being'"'"'s image to a plurality of templates based on matching at least one pixel of the being'"'"'s image upon its corresponding pixel in each of the templates, to generate a plurality of template outputs; and

analyzing the template outputs by

identifying at least one general pre-defined gesture;

identifying at least one specific pre-defined gesture in the identified general pre-defined gesture such that the specific pre-defined gesture corresponds to the gesture in the image; and

repeating from the step of retrieving the image to recognize another gesture based on the same background image.

16. A method as recited in claim 15 wherein:

the step of identifying at least one general pre-defined gesture selects at least one approach out of a neural network approach and a rule-based approach; and

the step of identifying at least one specific pre-defined gesture selects at least one approach out of a neural network approach and a rule-based approach.

17. A method as recited in claim 16 wherein the step of identifying at least one specific pre-defined gesture depends on at least the outputs from the step of identifying at least one general pre-defined gesture.

18. A method as recited in claim 16 wherein the step of identifying at least one specific pre-defined gesture further includes the step of applying a set of rules to the output of the neural network if the neural network approach has been selected.

19. A method for recognizing the gesture of the image of a being by identifying a gesture from a set of specific pre-defined gestures, with the image having a plurality of pixels, where every pixel has a value, the method comprising the steps of:

retrieving a background image;

retrieving a current image that includes the being'"'"'s image and at least a portion of the background image;

retrieving a threshold value;

retrieving a upper limit value;

removing at least a portion of the background image from the current image to generate the being'"'"'s image by;

comparing the difference between the value of each pixel in the background image and the value of the corresponding pixel in the current image with the threshold value to generate an energy level;

changing the threshold value if the energy level is higher than the upper limit value;

repeating from the step of comparing if the energy level is higher than the upper limit value; and

generating the being'"'"'s image based on the background image, the current image and the threshold value;

mapping the being'"'"'s image to a plurality of templates to generate a plurality of template outputs, based on matching at least one pixel of the being'"'"'s image upon its corresponding pixel in each of the templates;

analyzing the template outputs to identify the specific pre-defined gesture that corresponds to the gesture in the image of the being; and

repeating from the step of retrieving a current image to recognize another gesture based on the same background image.

20. A method as recited in claim 19 wherein:

the captured images are gray-scale images with a plurality of pixels;

the step of comparing the difference comprises the steps of;

subtracting the value of each pixel in the background image from the value of the corresponding pixel in the current image to form a difference image;

for every pixel in the difference image;

if the magnitude of its value is below the threshold value, set that value to zero;

else setting that value to one to form the being'"'"'s image; and

adding the values of all of the pixels in the being'"'"'s image to form the energy level; and

the step of changing the threshold level comprises the steps of increasing the threshold value by a delta if the energy level is higher than the upper limit value.

21. A method as recited in claim 19:

further comprising the step of retrieving a lower limit value after the step of retrieving a upper limit vale; and

wherein;

the step of changing the threshold value further comprises the step of changing the threshold value if the energy level is lower than the lower limit value; and

the step of repeating from the step of comparing further comprises the step of repeating from the step of comparing if the energy level is lower than the lower imit value.

22. A method for recognizing the gesture of the image of a being by identifying a gesture from a set of specific pre-defined gestures, the method comprising the steps of:

retrieving a background image;

retrieving a current image that includes the being'"'"'s image and at least a portion of the background image;

removing at least a portion of the background image form the current image to generate the being'"'"'s image, which has a plurality of pixels;

mapping the being'"'"'s image to a plurality of templates based on matching at least one pixel of the being'"'"'s image upon its corresponding pixel in each of the templates, to generate a plurality of template outputs;

with each template being represented by a hologram; and

with the template outputs being generated by optically correlating the being'"'"'s image with the plurality of holograms;

analyzing the template outputs to identify the specific pre-defined gesture that corresponds to the gesture in the image; and

repeating from the step of retrieving a current image to recognize another gesture based on the same background image.

23. A method as recited in claim 22 wherein all of the holograms reside substantially in the same area of a holographic medium.

24. A method as recited in claim 22 wherein:

the holograms reside in different locations of a holographic medium;

the being'"'"'s image is duplicated into a number of duplicated images; and

each duplicated image is directed to one hologram to perform optical correlation.

25. A method for recognizing the gesture of the image of a being by identifying a gesture from a set of specific pre-defined gestures, with the image having a plurality of pixels, where every pixel has a value, the method comprising the steps of:

retrieving the being'"'"'s image, which is digitized, with the being'"'"'s image generated based on a background image;

retrieving a plurality of templates, which are digitized;

digitally compressing the being'"'"'s image to a compressed being'"'"'s image;

digitally compressing the templates to a plurality of compressed templates; and

mapping the compressed being'"'"'s image to the plurality of compressed templates based on matching at least one pixel of the being'"'"'s image upon its corresponding pixel in each of the templates, to generate a plurality of template outputs, with each template output being generated by the process that includes the step of performing an inner product of the corresponding compressed template with the compressed being'"'"'s image;

analyzing the template outputs to identify the specific pre-defined gesture that corresponds to the gesture in the image; and

repeating from the step of retrieving the being'"'"'s image to recognize another gesture based on the same background image.

26. A method as recited in claim 25 wherein:

the being'"'"'s image is binarized;

the plurality of templates are binarized; and

at least one inner product performs the AND operation between the compressed being'"'"'s image and a compressed template.

27. A method as recited in claim 26 wherein one inner product uses a look-up table.

28. A method for recognizing the gesture of the image of a being by identifying a gesture from a set of specific pre-defined gestures, with the image having a plurality of pixels, where every pixel has a value, the method comprising the steps of:

retrieving a background image;

retrieving a current image that includes the being'"'"'s image and at least a portion of the background image;

removing a least a portion of the background image from the current image to generate the being'"'"'s image, which includes a plurality of pixels;

mapping the being'"'"'s image to a plurality of templates based on matching at least one pixel of the being'"'"'s image upon its corresponding pixel in each of the templates, to generate a plurality of template outputs;

analyzing the template outputs by selecting at least one approach out of the neural network approach and the rule-based approach to identify the specific pre-defined gesture that corresponds to the gesture in the image; and

repeating from the step of retrieving a current image to recognize another gesture based on the same background image.

29. A method as recited in claim 28 wherein the neural network is a multi-level neural network.

30. A method as recited in claim 28 wherein:

the set of specific pre-defined gestures are classified into one or more general pre-defined gestures;

the step of analyzing includes the steps of;

entering the template outputs into the neural network to generate a set of neural outputs representing at least one general pre-defined gesture that the gesture of the image should be categorized as; and

applying a set of rules to the neural outputs to identify the specific pre-defined gesture that corresponds to the gesture in the image.

31. A method as recited in claim 28 wherein:

the set of specific pre-defined gestures are classified into one or more general pre-defined gestures;

the neural network includes a first level neural network and a second level neural network; and

the step of analyzing includes the steps of;

entering the template outputs into the first level neural network to generate a first set of neural outputs, which represent at least one general pre-defined gesture that the gesture of the image should be classified as; and

entering at least the first set of neural outputs into the second level neural network to identify the specific pre-defined gesture that corresponds to the gesture in the image.

32. A method as recited in claim 31 wherein the step of analyzing further includes the step of applying a set of rules to the output of the second level neural network to identify the specific pre-defined gesture that corresponds to the gesture in the image.

33. A method for recognizing the gesture of the image of a being by identifying a gesture from a set of specific pre-defined gestures, the method comprising the steps of:

retrieving the image of the being generated based on a background image, with the being'"'"'s image having a plurality of pixels;

mapping the being'"'"'s image to at least one set of a plurality of sets of templates based on matching at least one pixel of the being'"'"'s image upon its corresponding pixel in each of the templates, to generate a plurality of template outputs;

identifying at least one specific position on the being'"'"'s image based on the template outputs;

repeating from the step of retrieving based on the same background image to identify the change in the specific position of the being'"'"'s image;

analyzing the template outputs based on the change in the specific position of the being'"'"'s image to identify the specific pre-defined gesture that corresponds to the gesture of the image.

34. A method as recited in claim 33 wherein:

the specific position is the center of the being'"'"'s image;

the steps of mapping includes the step of mapping the being'"'"'s image to each template to generate a plurality of template outputs; and

the method further comprises the step of deleting at least one template output based on the change in the position of the center of the being'"'"'s image before the step of analyzing the template outputs.

35. A method as recited in claim 33 wherein before the step of analyzing, the method further comprises the steps of:

deleting at least one template from one set of templates that has not been mapped to the being'"'"'s image, based on the change in the specific position of the being'"'"'s image; and

mapping the being'"'"'s image to the templates that have not been mapped before to generate additional template outputs.