Gesture cataloging and recognition

• US 8,225,343 B2
• Filed: 12/17/2008
• Issued: 07/17/2012
• Est. Priority Date: 01/11/2008
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment America LLC (Sony Group Corp.)
• Inventors: Miller, Thomas M. IV
• Primary Examiner(s): Bui, Kieu Oanh T
• Application Number:
  US12/337,477
• Publication Number:
  US 20090183193A1
• Time in Patent Office: 1,308 Days
• Field of Search: 725/10, 725/12, 725/133, 725/141, 725/153, 463 37- 39, 345/158
• US Class Current: 725/12
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Methods and apparatus for cataloging and recognizing gestures are disclosed. A gesture may be detected using sample motion data. An energy value and a baseline value may be computed. The baseline value may be updated if the energy value is below a calm energy threshold. The sample motion data may be adjusted based on the updated baseline value. A local variance may be calculated over a predetermined number of samples. Sample motion data values may be recorded if the local variance exceeds a threshold. Sample motion data recording may stop if a local variance scalar value falls below a drop threshold. Input Gestures may be recognized by computing a total variance for sample values in an Input Gesture; calculating a figure of merit using sample values from the Input Gesture and one or more Catalog Gestures; and determining whether the Input Gesture matches a Catalog Gesture from the figure of merit.

Claims

1. A method for gesture recognition, comprising:

a) receiving sample motion data from one or more sensors associated with a control device wherein the motion data is related to movement of the control device;

b) computing an energy value from the motion data and a baseline value for the motion data;

c) updating the baseline value based on the energy value if the energy value is less than a calm energy threshold;

d) adjusting the sample motion data based on the updated baseline value;

e) calculating a local variance of the sample motion data over a predetermined number of local variance samples;

f) beginning recording one or more values of the sample motion data for a gesture if the local variance scalar value is greater than a threshold for beginning recording;

g) calculating an average local variance scalar value using the one or more values of the sample motion data for a gesture; and

h) stopping the recording of the one or more values of the sample motion data for a gesture if the local variance scalar value is less than a threshold for stopping recording.

2. The method of claim 1, further comprising:

i) computing a total variance for sample values in the gesture;

j) calculating a figure of merit using the sample values in the gesture and sample values in one or more catalog gesture, wherein the figure of merit is a measure of how well the gesture matched the catalog gesture;

k) determining whether an input gesture matches one of the one or more catalog gesture based on the figure of merit; and

l) taking action if the input gesture matches the one of the one or more catalog gesture.

3. The method of claim 2, wherein the sample motion data comprises vector components.

4. The method of claim 3, wherein the total variance is calculated with ranks assigned to components of the total variance corresponding to different ones of the vector components.

5. The method of claim 4, wherein j) includes comparing input gesture component ranks to catalog gesture component ranks.

6. The method of claim 5, wherein j) includes weighting components by ranking of total variance components.

7. The method of claim 6, wherein j) includes calculating a correlation between the input gesture and the catalog gesture.

8. The method of claim 6, further comprising scaling up the input gesture or catalog gesture based on which has fewer samples.

9. The method of claim 6, further comprising comparing total variance rankings for input gesture and catalog gesture and applying a penalty for each of input gesture rank that does not match.

10. The method of claim 9, wherein the figure of merit comprise a correlation scalar calculated as weighted average of scalar correlation with weights based on the total variance rankings.

11. The method of claim 10, wherein total variance rankings are calculated at different time intervals within the gesture.

12. The method of claim 11, wherein weights may be different at different intervals of time within the gesture.

13. The method of claim 2, wherein the method is applied to a video game.

14. The method of claim 13, wherein taking action comprises responding to command associated with the gesture.

15. The method of claim 14, wherein comparing input gesture to selected subject of catalog gesture based on the game logic.

16. The method of claim 1, wherein the sample motion data includes data derived from a signal from an inertial sensor.

17. The method of claim 1, wherein the sample motion data includes data from an image from an image capturing device.

18. The method of claim 1, wherein the sample motion data includes data derived from one or more signals obtained from one or more acoustic sensors.

19. An apparatus for gesture recognition comprising:

a control device;

a processor operable to execute a program of a method for gesture recognition, wherein the method comprising;

receiving sample motion data from one or more sensors associated with the control device wherein the motion data is related to movement of the control device;

computing an energy value from the motion data and a baseline value for the motion data;

updating the baseline value based on the energy value if the energy value is less than a calm energy threshold;

adjusting the sample motion data based on the updated baseline value;

calculating a local variance if the sample motion data over a predetermined number of local variance samples;

beginning recording one or more values of the sample motion data for a gesture if the local variance scalar value is greater than a start threshold;

calculating an average local variance scalar value using the one or more values of the sample motion data for a gesture; and

stopping the recording of the one or more values of the sample motion data for a gesture if the local variance scalar value is less than a stop threshold.

20. The apparatus of claim 19, wherein the method for gesture recognition further comprises:

computing a total variance for sample values in the gesture;

calculating a figure of merit using the sample values in the gesture and sample values in one or more catalog gesture, wherein the figure of merit is a measure of how well the gesture matched the catalog gesture;

determining whether an input gesture matches one of the one or more catalog gesture based on the figure of merit; and

taking action if the input gesture matches the one of the one or more catalog gesture.

21. A non-transitory computer readable storage medium having computer readable instructions embodied therein, the computer readable instructions being configured to implement, when executed, a method for gesture recognition, the method comprising:

receiving sample motion data from a sensor associated with a control device wherein the motion data is related to movement of the control device;

computing an energy value from the motion data and a baseline value for the motion data;

updating the baseline value based on the energy value if the energy value is less than a calm energy threshold;

adjusting the sample motion data based on the updated baseline value;

calculating a local variance if the sample motion data over a predetermined number of local variance samples;

beginning recording one or more values of the sample motion data for a gesture if the local variance scalar value is greater than a start threshold;

calculating an average local variance scalar value using the one or more values of the sample motion data for a gesture; and

stopping the recording of the one or more values of the sample motion data for a gesture if the local variance scalar value is less than a stop threshold.