Systems, devices, and methods for gesture identification

• US 10,684,692 B2
• Filed: 12/22/2017
• Issued: 06/16/2020
• Est. Priority Date: 06/19/2014
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Facebook Technologies, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Aleem, Idris S., Ataee, Pedram
• Primary Examiner(s): Chang, Kent W
• Assistant Examiner(s): Bibbee, Chayce R
• Application Number:
  US15/852,196
• Publication Number:
  US 20180120948A1
• Time in Patent Office: 907 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Systems, devices, and methods adapt established concepts from natural language processing for use in gesture identification algorithms. A gesture identification system includes sensors, a processor, and a non-transitory processor-readable memory that stores data and/or instructions for performing gesture identification. A gesture identification system may include a wearable gesture identification device. The gesture identification process involves segmenting signals from the sensors into data windows, assigning a respective “window class” to each data window, and identifying a user-performed gesture based on the corresponding sequence of window classes. Each window class exclusively characterizes at least one data window property and is analogous to a “letter” of an alphabet. Under this model, each gesture is analogous to a “word” made up of a particular combination of window classes.

Claims

1. A method of operating a gesture identification system to identify a user-performed gesture, the gesture identification system including at least one sensor responsive to user-performed gestures and a processor communicatively coupled to the at least one sensor, the method comprising:

providing at least one signal from the at least one sensor to the processor;

segmenting the at least one signal into data windows;

for each i^th data window in at least a subset of the data windows;

determining a window class for the i^th data window by the processor, the window class selected by the processor from a library of window classes, wherein each window class in the library of window classes exclusively characterizes at least one data window property;

determining, by the processor, a respective probability that each gesture in a gesture library is the user-performed gesture based on the window class for the i^th data window, wherein determining, by the processor, a respective probability that each gesture in a gesture library is the user-performed gesture includes;

determining, by the processor, at least one respective transition model for each gesture in the gesture library; and

determining, by the processor and for each gesture in the gesture library, a probability that the gesture is the user-performed gesture based at least in part on the at least one transition model for the gesture; and

identifying a highest-probability gesture for the i^th data window by the processor, the highest-probability gesture corresponding to the gesture in the gesture library that has a highest probability of being the user-performed gesture for the i^th data window; and

identifying the user-performed gesture by the processor based on the highest-probability gesture for at least one data window in the at least a subset of data windows.

2. The method of claim 1 wherein providing at least one signal from the at least one sensor to the processor includes providing at least one substantially continuous data stream from the at least one sensor to the processor, and wherein segmenting the at least one signal into data windows includes segmenting the at least one substantially continuous data stream into data windows in real-time by the processor.

3. The method of claim 1 wherein each window class in the library of window classes exclusively characterizes a respective range of values for the same at least one data window property.

4. The method of claim 3 wherein each window class in the library of window classes exclusively characterizes a respective range of values for at least one Root Mean Square (“

RMS”

) value for the i^th data window.

5. The method of claim 1 wherein the at least one sensor includes a plurality of sensors, and wherein:

providing at least one signal from the at least one sensor to the processor includes providing a respective signal from each respective sensor in the plurality of sensors to the processor; and

segmenting the at least one signal into data windows includes segmenting the respective signal from each respective sensor in the plurality of sensors into the data windows, wherein each data window includes a respective portion of the signal from each respective sensor in the plurality of sensors.

6. The method of claim 5 wherein the plurality of sensors includes N sensors, and wherein each window class in the library of window classes exclusively characterizes a respective region in an N-dimensional hyperspace and each dimension of the N-dimensional hyperspace represents a property of the signal from a respective one of the N sensors.

7. The method of claim 6 wherein each region of the N-dimensional hyperspace represents a respective combination of ranges for Root Mean Square (“

RMS”

) values of the signals from the N sensors.

8. The method of claim 6 wherein, for each window class in the library of window classes, the corresponding region in the N-dimensional hyperspace is exclusively characterized by at least one respective angle formed in the N-dimensional hyperspace.

9. The method of claim 1 wherein the at least one sensor includes at least one muscle activity sensor selected from the group consisting of:

an electromyography (EMG) sensor and a mechanomyography (MMG) sensor, and wherein;

providing at least one signal from the at least one sensor to the processor includes providing at least one signal from the at least one muscle activity sensor to the processor.

10. The method of claim 1 wherein the at least one sensor includes at least one inertial sensor selected from the group consisting of:

an accelerometer, a gyroscope, and an inertial measurement unit (IMU), and wherein;

providing at least one signal from the at least one sensor to the processor includes providing at least one signal from the at least one inertial sensor to the processor.

11. The method of claim 1 wherein the gesture identification system further comprises a non-transitory processor-readable storage medium that stores processor-executable gesture identification instructions, the non-transitory processor-readable storage medium communicatively coupled to the processor, and the method further comprising:

executing by the processor the gesture identification instructions to cause the processor to;

determine the window class for each i^th data window;

determine the respective probability that each gesture in the gesture library is the user-performed gesture for each i^th data window;

identify the highest-probability gesture for each i^th data window; and

identify the user-performed gesture.

38. The method of claim 1 wherein determining, by the processor, at least one respective transition model for each gesture in the gesture library includes determining, by the processor and for each gesture in the gesture library, at least one respective transition model selected from the group consisting of:

a unigram transition model based on the window class of the i^th data window and a bigram transition model based on the respective window classes of the i^th data window and a j^th=(i−

1)^th data window.

12. A gesture identification system comprising:

at least one sensor responsive to physical gestures performed by a user of the gesture identification system, wherein in response to a physical gesture performed by the user the at least one sensor provides at least one signal;

a processor communicatively coupled to the at least one sensor; and

a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable gesture identification instructions that, when executed by the processor, cause the gesture identification device to;

segment the at least one signal into data windows;

for each i^th data window in at least a subset of the data windows;

determine a window class for the i^th data window, the window class selected from a library of window classes, wherein each window class in the library of window classes exclusively characterizes at least one data window property;

determine a respective probability that each gesture in a gesture library is the physical gesture performed by the user based on the window class for the i^th data window, wherein the processor-executable gesture identification instructions that, when executed by the processor, cause the gesture identification device to determine a respective probability that each gesture in a gesture library is the physical gesture performed by the user, cause the gesture identification device to;

determine at least one respective transition model for each gesture in the gesture library; and

determine, for each gesture in the gesture library, a probability that the gesture is the physical gesture performed by the user based at least in part on the at least one transition model for the gesture; and

identify a highest-probability gesture for the i^th data window, the highest-probability gesture corresponding to the gesture in the gesture library that has a highest probability of being the physical gesture performed by the user for the i^th data window; and

identify the physical gesture performed by the user based on the highest-probability gesture for at least one data window in the at least a subset of data windows.

13. The gesture identification system of claim 12 wherein the at least one sensor includes a sensor selected from the group consisting of:

a muscle activity sensor, an electromyography (EMG) sensor, a mechanomyography (MMG) sensor, an inertial sensor, an accelerometer, a gyroscope, an inertial measurement unit (IMU), a photosensor, an optical sensor, and an infrared sensor.

14. The gesture identification system of claim 12, further comprising:

a wearable gesture identification device comprising a band that in use is worn on a limb of the user, wherein the at least one sensor, the processor, and the non-transitory processor-readable storage medium are all carried by the band.

15. The gesture identification system of claim 12 wherein each window class in the library of window classes exclusively characterizes a respective range of values for the same at least one data window property.

16. The gesture identification system of claim 12 wherein the at least one sensor includes a plurality of sensors, each of which is responsive to physical gestures performed by the user, and wherein the processor-executable gesture identification instructions, when executed by the processor, cause the gesture identification device to:

segment the respective signal from each respective sensor in the plurality of sensors into the data windows, wherein each data window includes a respective portion of the signal from each respective sensor in the plurality of sensors.

17. The gesture identification system of claim 12 wherein the at least one sensor includes N sensors, when N≥

1, and wherein each window class in the library of window classes exclusively characterizes a respective region in an N-dimensional hyperspace, each dimension of the N-dimensional hyperspace representing a property of the signal from a respective one of the N sensors.

18. The gesture identification system of claim 17 wherein each region of the N-dimensional hyperspace represents a respective combination of ranges for Root Mean Square (“

RMS”

) values of the signals from the N sensors.

19. The gesture identification system of claim 17 wherein, for each window class in the library of window classes, the corresponding region in the N-dimensional hyperspace is exclusively characterized by at least one respective angle formed in the N-dimensional hyperspace.

39. The gesture identification system of claim 12 wherein determining at least one respective transition model for each gesture in the gesture library includes determining, for each gesture in the gesture library, at least one respective transition model selected from the group consisting of:

a unigram transition model based on the window class of the i^th data window and a bigram transition model based on the respective window classes of the i^th data window and a j^th=(i−

1)^th data window.

20. A method of operating a gesture identification system to identify a user-performed gesture, the gesture identification system including at least one sensor responsive to user-performed gestures and a processor communicatively coupled to the at least one sensor, the method comprising:

in response to a user-performed gesture, providing at least one signal from the at least one sensor to the processor;

segmenting the at least one signal into data windows;

for each data window in at least a subset of the data windows, assigning a window class to the data window by the processor, each respective window class selected by the processor from a library of window classes, wherein each window class in the library of window classes exclusively characterizes at least one data window property;

determining, by the processor, a respective probability that each gesture in a gesture library is the user-performed gesture based on the window class of at least one data window, wherein determining, by the processor, a respective probability that each gesture in a gesture library is the user-performed gesture includes;

determining, by the processor and for each gesture in the gesture library, at least one respective transition model based on the window class of a single data window; and

determining, by the processor and for each gesture in the gesture library, a probability that the gesture is the user-performed gesture based at least in part on the at least one transition model for the gesture;

identifying the user-performed gesture by the processor based at least in part on the window class of at least one data window, wherein identifying the user-performed gesture by the processor includes identifying, by the processor, a highest-probability gesture, the highest-probability gesture corresponding to a gesture in the gesture library that has a highest probability of being the user-performed gesture.

21. The method of claim 20 wherein the at least one sensor includes N sensors, where N≥

1, and wherein each window class in the library of window classes exclusively characterizes a respective region in an N-dimensional hyperspace and each dimension of the N-dimensional hyperspace represents a property of the signal from a respective one of the N sensors.

22. The method of claim 21 wherein each region of the N-dimensional hyperspace represents a respective combination of ranges for Root Mean Square (“

RMS”

) values of the signals from the N sensors.

23. The method of claim 21 wherein, for each window class in the library of window classes, the corresponding region in the N-dimensional hyperspace is exclusively characterized by at least one respective angle formed in the N-dimensional hyperspace.

24. The method of claim 20 wherein providing at least one signal from the at least one sensor to the processor includes providing at least one substantially continuous data stream from the at least one sensor to the processor, and wherein segmenting the at least one signal into data windows includes segmenting the at least one substantially continuous data stream into data windows in real-time by the processor.

25. The method of claim 20 wherein each window class in the library of window classes exclusively characterizes a respective range of values for the same at least one data window property.

26. The method of claim 25 wherein each window class in the library of window classes exclusively characterizes a respective range of values for at least one data window Root Mean Square (“

RMS”

) value.

27. The method of claim 20 wherein the at least one sensor includes a plurality of sensors, and wherein:

providing at least one signal from the at least one sensor to the processor includes providing a respective signal from each respective sensor in the plurality of sensors to the processor; and

segmenting the at least one signal into data windows includes segmenting the respective signal from each respective sensor in the plurality of sensors into the data windows, wherein each data window includes a respective portion of the signal from each respective sensor in the plurality of sensors.

28. The method of claim 20 wherein the at least one sensor includes at least one muscle activity sensor selected from the group consisting of:

an electromyography (EMG) sensor and a mechanomyography (MMG) sensor, and wherein;

providing at least one signal from the at least one sensor to the processor includes providing at least one signal from the at least one muscle activity sensor to the processor.

29. The method of claim 20 wherein the at least one sensor includes at least one inertial sensor selected from the group consisting of:

an accelerometer, a gyroscope, and an inertial measurement unit (IMU), and wherein;

providing at least one signal from the at least one sensor to the processor includes providing at least one signal from the at least one inertial sensor to the processor.

30. The method of claim 20 wherein the gesture identification system further comprises a non-transitory processor-readable storage medium that stores processor-executable gesture identification instructions, the non-transitory processor-readable storage medium communicatively coupled to the processor, and the method further comprising:

executing by the processor the gesture identification instructions to cause the processor to;

for each data window in at least a subset of the data windows, assign the window class to the data window; and

identify the user-performed gesture based at least in part on the window class of at least one data window.

31. A gesture identification system comprising:

at least one sensor responsive to physical gestures performed by a user of the gesture identification system, wherein in response to a physical gesture performed by the user the at least one sensor provides at least one signal;

a processor communicatively coupled to the at least one sensor; and

a non-transitory processor-readable storage medium communicatively coupled to the processor, wherein the non-transitory processor-readable storage medium stores processor-executable gesture identification instructions that, when executed by the processor, cause the gesture identification device to;

segment the at least one signal into data windows;

for each data window in at least a subset of the data windows, assign a window class to the data window;

determine a respective probability that each gesture in a gesture library is the physical gesture performed by the user based on the window class of at least one data window, wherein the processor-executable gesture identification instructions that, when executed by the processor, cause the gesture identification device to determine a respective probability that each gesture in a gesture library is the physical gesture performed by the user, cause the gesture identification device to;

determine, for each gesture in the gesture library, at least one respective transition model based on the window class of a single data window; and

determine, for each gesture in the gesture library, a probability that the gesture is the physical gesture performed by the user based at least in part on the at least one transition model for the gesture; and

identify the physical gesture performed by the user based at least in part on the window class of at least one data window, wherein the processor-executable gesture identification instructions that, when executed by the processor, cause the gesture identification device to identify the physical gesture performed by the user based at least in part on the window class of at least one data window, cause the gesture identification device to identify a highest-probability gesture, the highest-probability gesture corresponding to a gesture in the gesture library that has a highest probability of being the physical gesture performed by the user.

32. The gesture identification system of claim 31 wherein the at least one sensor includes a sensor selected from the group consisting of:

a muscle activity sensor, an electromyography (EMG) sensor, a mechanomyography (MMG) sensor, an inertial sensor, an accelerometer, a gyroscope, an inertial measurement unit (IMU), a photosensor, an optical sensor, and an infrared sensor.

33. The gesture identification system of claim 31, further comprising:

a wearable gesture identification device comprising a band that in use is worn on a limb of the user, wherein the at least one sensor, the processor, and the non-transitory processor-readable storage medium are all carried by the band.

34. The gesture identification system of claim 31 wherein each window class in the library of window classes exclusively characterizes a respective range of values for the same at least one data window property.

35. The gesture identification system of claim 31 wherein the at least one sensor includes N sensors, where N≥

1, and wherein each window class in the library of window classes exclusively characterizes a respective region in an N-dimensional hyperspace and each dimension of the N-dimensional hyperspace represents a property of the signal from a respective one of the N sensors.

36. The gesture identification system of claim 35 wherein each region of the N-dimensional hyperspace represents a respective combination of ranges for Root Mean Square (“

RMS”

) values of the signals from the N sensors.

37. The gesture identification system of claim 35 wherein, for each window class in the library of window classes, the corresponding region in the N-dimensional hyperspace is exclusively characterized by at least one respective angle formed in the N-dimensional hyperspace.