CLASSIFICATION OF GESTURE DETECTION SYSTEMS THROUGH USE OF KNOWN AND YET TO BE WORN SENSORS

• US 20150091790A1
• Filed: 09/30/2013
• Published: 04/02/2015
• Est. Priority Date: 09/30/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Forutanpour, Babak, Wenger, Geoffrey C., Carvalho, Vitor R.
• Granted Patent: US 10,048,761 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser

Abstract

An apparatus, a method, and a computer program product for gesture recognition. The apparatus classifies a gesture based on a movement of a body part as detected by a primary sensor. The apparatus determine a reliability level of a secondary sensor and obtains corroborating information associated with the movement of the body part using the secondary sensor when the reliability level satisfies a criterion. The apparatus then confirms or negates the classification of the gesture based on the corroborating information. The secondary sensor may be a sensor already known to the apparatus, i.e., the sensor is currently being worn by the user, or it may be a sensor that is worn by a user at a later time. In the latter case, the apparatus detects for the presence of a new sensor, determines the gesture recognition capabilities of the new sensor and integrates the new sensor into the gesture recognition process.

Claims

1. A method of gesture recognition, comprising:

classifying a gesture based on a movement of a body part, the movement detected by a primary sensor;

determining a reliability level of a secondary sensor;

when the reliability level satisfies a criterion, obtaining corroborating information associated with the movement of the body part using the secondary sensor; and

confirming or negating the classification of the gesture based on the corroborating information.

2. The method of claim 1, wherein determining a reliability level of a secondary sensor comprises measuring an environmental condition associated with the secondary sensor.

3. The method of claim 2, wherein the environmental condition comprises one or more of a sound level or a light level.

4. The method of claim 1, wherein the corroborating information comprises one or more of an image of the body part or a sound emanating from the body part.

5. The method of claim 1, wherein the primary sensor and the secondary sensor are different types of sensors.

6. The method of claim 1, wherein confirming or negating the classification of the gesture comprises:

determining a corroborating gesture based on the corroborating information; and

confirming the classification of the gesture when the corroborating gesture matches the classification of the gesture.

7. The method of claim 6, wherein determining a corroborating gesture comprises comparing the corroborating information to corresponding information mapped to a library of gestures.

8. The method of claim 1, wherein classifying a gesture based on movement of a body part as detected by a primary sensor comprises:

sensing a motion activity resulting from the movement of the body part;

comparing the sensed motion activity to one or more corresponding stored motion activities mapped to a library of gestures; and

concluding the body part made the gesture when the sensed motion activity matches the stored motion activity mapped to the gesture.

9. The method of claim 8, further comprising updating the library of gestures when the classification of the gesture is negated based on the corroborating information.

10. The method of claim 9, wherein updating the library of gestures comprises remapping the stored motion activity corresponding to the sensed motion activity to a different gesture, wherein the different gesture corresponds to a corroborating gesture determined based on the corroborating information.

11. The method of claim 1, wherein the gesture has an associated confidence level, the method further comprising:

comparing the confidence level to a threshold; and

performing the determining, the obtaining and the confirming or negating only when the confidence level does not satisfy the threshold.

12. The method of claim 1, further comprising detecting for the presence of the secondary sensor prior to performing the determining, the obtaining and the confirming or negating.

13. The method of claim 1, further comprising:

monitoring data from the secondary sensor provided during the movement of the body part; and

assessing reliability of data provided by the primary sensor based on data provided by the secondary sensor.

14. An apparatus for gesture recognition, said apparatus comprising:

means for classifying a gesture based on a movement of a body part, the movement detected by a primary sensor;

means for determining a reliability level of a secondary sensor;

means for obtaining corroborating information associated with the movement of the body part using the secondary sensor when the reliability level satisfies a criterion; and

means for confirming or negating the classification of the gesture based on the corroborating information.

15. The apparatus of claim 14, wherein the means for determining a reliability level of a secondary sensor is configured to measure an environmental condition associated with the secondary sensor.

16. The apparatus of claim 15, wherein the environmental condition comprises one or more of a sound level or a light level.

17. The apparatus of claim 14, wherein the corroborating information comprises one or more of an image of the body part or a sound emanating from the body part.

18. The apparatus of claim 14, wherein the primary sensor and the secondary sensor are different types of sensors.

19. The apparatus of claim 14, wherein the means for confirming or negating the classification of the gesture is configured to:

determine a corroborating gesture based on the corroborating information; and

confirm the classification of the gesture when the corroborating gesture matches the classification of the gesture.

20. The apparatus of claim 19, wherein the means for confirming or negating the classification of the gesture determines a corroborating gesture by being further configured to compare the corroborating information to corresponding information mapped to a library of gestures in order to determine a corroborating gesture.

21. The apparatus of claim 14, wherein the means for classifying a gesture based on movement of a body part as detected by a primary sensor is configured to:

sense a motion activity resulting from the movement of the body part;

compare the sensed motion activity to one or more corresponding stored motion activities mapped to a library of gestures; and

conclude the body part made the gesture when the sensed motion activity matches the stored motion activity mapped to the gesture.

22. The apparatus of claim 21, wherein the means for classifying a gesture is configured to update the library of gestures when the classification of the gesture is negated based on the corroborating information.

23. The apparatus of claim 22, wherein the means for classifying a gesture updates the library of gestures by being further configured to remap the stored motion activity corresponding to the sensed motion activity to a different gesture, wherein the different gesture corresponds to a corroborating gesture determined based on the corroborating information.

24. The apparatus of claim 14, wherein the gesture has an associated confidence level, the apparatus further comprising:

means for comparing the confidence level to a threshold; and

means for performing the determining, the obtaining and the confirming or negating only when the confidence level does not satisfy the threshold.

25. The apparatus of claim 14, further comprising means for detecting for the presence of the secondary sensor prior to performing the determining, the obtaining and the confirming or negating.

26. The apparatus of claim 14, further comprising:

means for monitoring data from the secondary sensor provided during the movement of the body part; and

means for assessing reliability of data provided by the primary sensor based on data provided by the secondary sensor.

27. An apparatus for gesture recognition, said apparatus comprising:

a memory; and

a processing system coupled to the memory and configured to;

classify a gesture based on a movement of a body part, the movement detected by a primary sensor;

determine a reliability level of a secondary sensor;

obtain corroborating information associated with the movement of the body part using the secondary sensor when the reliability level satisfies a criterion; and

confirm or negate the classification of the gesture based on the corroborating information.

28. The apparatus of claim 27, wherein the processing system determine a reliability level of a secondary sensor by being configured to measuring an environmental condition associated with the secondary sensor.

29. The apparatus of claim 28, wherein the environmental condition comprises one or more of a sound level or a light level.

30. The apparatus of claim 27, wherein the corroborating information comprises one or more of an image of the body part or a sound emanating from the body part.

31. The apparatus of claim 27, wherein the primary sensor and the secondary sensor are different types of sensors.

32. The apparatus of claim 27, wherein the processing system confirms or negates the classification of the gesture by being further configured to:

determine a corroborating gesture based on the corroborating information; and

confirm the classification of the gesture when the corroborating gesture matches the classification of the gesture.

33. The apparatus of claim 32, wherein the processing system determines a corroborating gesture by being configured to compare the corroborating information to corresponding information mapped to a library of gestures.

34. The apparatus of claim 27, wherein the processing system classifies a gesture based on movement of a body part as detected by a primary sensor by being configured to:

sense a motion activity resulting from the movement of the body part;

compare the sensed motion activity to one or more corresponding stored motion activities mapped to a library of gestures; and

conclude the body part made the gesture when the sensed motion activity matches the stored motion activity mapped to the gesture.

35. The apparatus of claim 34, wherein the processing system is further configured to update the library of gestures when the classification of the gesture is negated based on the corroborating information.

36. The apparatus of claim 35, wherein the processing system updates the library of gestures by being further configured to remap the stored motion activity corresponding to the sensed motion activity to a different gesture, wherein the different gesture corresponds to a corroborating gesture determined based on the corroborating information.

37. The apparatus of claim 27, wherein the gesture has an associated confidence level, the processing system being further configured to:

compare the confidence level to a threshold; and

perform the determining, the obtaining and the confirming or negating only when the confidence level does not satisfy the threshold.

38. The apparatus of claim 27, wherein the processing system is further configured to detect for the presence of the secondary sensor prior to performing the determining, the obtaining and the confirming or negating.

39. The apparatus of claim 27, wherein the processing system is further configured to:

monitor data from the secondary sensor provided during the movement of the body part; and

assess reliability of data provided by the primary sensor based on data provided by the secondary sensor.

40. A computer program product for gesture recognition, said product comprising:

a computer-readable medium comprising code for;

classifying a gesture based on a movement of a body part, the movement detected by a primary sensor;

determining a reliability level of a secondary sensor;

obtaining corroborating information associated with the movement of the body part using the secondary sensor when the reliability level satisfies a criterion; and

confirming or negating the classification of the gesture based on the corroborating information.