Trainable sensor-based gesture recognition

• US 9,405,377 B2
• Filed: 03/15/2014
• Issued: 08/02/2016
• Est. Priority Date: 03/15/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Li, MengKe
• Primary Examiner(s): NGUYEN, KIMNHUNG T
• Application Number:
  US14/214,707
• Publication Number:
  US 20150261298A1
• Time in Patent Office: 871 Days
• Field of Search: 345156-158, 345173-179, 345/204
• US Class Current: 1/1
• CPC Class Codes: G01H 17/00 Measuring mechanical vibrations or ultrasonic, sonic or infrasonic waves, not provided for in the preceding groups see provisionally also G01H1/00
• CPC Class Codes: G01P 15/00 Measuring acceleration; Measuring deceleration; Measuring shock, i.e. sudden change of acceleration
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

In many computing scenarios, a device comprises at least one sensor, and is configured to recognize a gesture performed by a user according to the sensor output of the sensor, and to perform a particular action upon recognizing the gesture. However, many devices are preconfigured with such gestures, and the recognition is specific to the sensors of the device, and is not specific to the manner in which a particular user performs the gesture. Presented herein are techniques for enabling a device to recognize a new gesture by monitoring the sensor output of any sensors provided by the device while the user performs the gesture, optionally requesting repeated gesture performances until reaching a recognition confidence. Once trained to recognize the gesture according to the sensor outputs of the particular sensors of the device, the device may subsequently recognize the gesture performed by the user and execute an associated action.

Claims

1. A method of performing an action associated with a gesture performed by a user of a device having a processor and a sensor providing a sensor output, the method comprising:

executing on the processor instructions that cause the device to;

transition the device to a training mode upon receiving, from the user, a training start, request;

during the training mode;

while the user performs the gesture, monitor the sensor to detect an identified sensor output identifying the gesture during a training period between the training start request and the training completion request;

identify, during the training period, a restricted period that is shorter than the training period and during which the identified sensor output identifies the gesture;

associate with the gesture the identified sensor output of the sensor during the restricted period; and

associate an action with the gesture;

transition the device to a recognition mode upon receiving, from the user, a training completion request; and

during the recognition mode;

monitor the sensor output of the sensor; and

upon detecting that the sensor output matches the identified sensor output associated with the gesture, perform the action associated with the gesture.

2. The method of claim 1, wherein executing the instructions on the processor further causes the device to, upon detecting an addition to the device of an added sensor, transition the device to the training mode to detect the identified sensor output of the added sensor while the user performs the gesture.

3. The method of claim 1, wherein executing the instructions on the processor further causes the device to, upon detecting a gesture that is not associated with an identified sensor output of a sensor within a recognition confidence threshold, transition the device to the training mode to retrain the sensor to recognize the gesture.

4. The method of claim 1, wherein executing the instructions on the processor further causes the device to, upon detecting a miscalibration of a sensor that reduces a recognition accuracy of the identified sensor output associated with the gesture, transition the device to the training mode to retrain the sensor to recognize the gesture.

5. The method of claim 1, wherein executing the instructions on the processor further causes the device to, upon receiving a gesture profile specifying a selected gesture that is associated with an identified sensor output that is not compatible with the sensor of the device, transition the device to the training mode to train the sensor to recognize the gesture.

6. The method of claim 1, wherein executing the instructions on the processor further causes the device to, upon detecting a sensor output of the sensor identifying a detected gesture that is not associated with an action, request the user to associate the detected gesture with at least one action.

7. The method of claim 1, wherein monitoring the sensor to detect the identified sensor output further comprises:

during respective at least two gesture performances of the gesture by the user, monitoring the sensor to detect the identified sensor output identifying the gesture during the gesture performance; and

determining the identified sensor output according to the identified sensor outputs of the respective at least two gesture performances.

8. The method of claim 7, wherein determining the identified sensor output further comprises:

comparing the identified sensor outputs during the respective at least two gesture performances to determine a recognition confidence;

while the recognition confidence is above a recognition confidence threshold, instructing the user to repeat the gesture performance; and

upon determining the recognition confidence within the recognition confidence threshold, associating the identified sensor output with the gesture.

9. The method of claim 1, wherein the association between the identified sensor output and the gesture is stored in a gesture profile, the method further comprising executing on the processor instructions that cause the device to send the gesture profile to a gesture profile service.

10. The method of claim 1, further comprising executing on the processor instructions that cause the device to retrieve a gesture profile from a gesture profile service, the gesture profile comprising an association of a gesture with an identified sensor output of at least one sensor and an action.

11. The method of claim 10, further comprising executing on the processor instructions that cause the device to transition the device to the training mode in response to detecting that the gesture profile'"'"'s identified sensor output of the at least one sensor comprises two or more sensor outputs of two or more sensors, at least one of which is absent from the device.

12. The method of claim 10, further comprising executing on the processor instructions that cause the device to:

detect that the gesture profile'"'"'s identified sensor output of the at least one sensor comprises two or more sensor outputs of two or more sensors, at least one of which is absent from the device; and

translate the at least one sensor output of the at least one sensor that is absent from the device into translated sensor output of the sensor that is part of the device.

13. A device that performs actions associated with gestures performed by a user, the device comprising:

a memory;

at least two sensors providing at least two sensor outputs;

a gesture trainer that;

while the user performs the gesture, monitors the at least two sensors to detect one or more identified sensor outputs identifying the gesture, the detecting the one or more identified sensor output comprises;

identifying, at least one recognizing sensor providing an identified sensor output identifying the gesture; and

identifying at least one non-recognizing sensor that does not provide an identified sensor output identifying the gesture; and

stores in the memory the one or more identified sensor outputs that together identify the gesture, the storing excluding, the identified sensor output of the at least one non-recognizing sensor;

an action associator that, upon receiving from the user an action to be associated with the gesture, stores in the memory an association of the action with the gesture; and

a gesture recognizer that;

monitors the at least two sensor outputs of the at least two sensors to detect the one or more identified sensor outputs identifying the gesture; and

performs the action associated in the memory with the gesture associated with the one or more identified sensor outputs.

14. The device of claim 13, wherein:

detecting the identified sensor output identifying the gesture further comprises;

requesting the user to identify, among the at least two sensors, at least one selected sensor identifying the gesture;

while the user performs the gesture, monitoring only the at least one selected sensor to detect an identified sensor output identifying the gesture; and

storing the one or more identified sensor outputs in the memory further comprises;

storing in the memory the identified sensor outputs only of the respective at least one selected sensor.

15. The device of claim 14, wherein:

the respective devices are associated with at least one sensor modality;

requesting the user to identify the at least one selected sensor further comprises;

requesting the user to identify, among at least two sensor modalities respectively associated with at least one sensor of the device, at least one selected sensor modality identifying the gesture;

detecting the identified sensor output further comprises;

while the user performs the gesture, monitoring only the at least one sensor associated with at least one selected sensor modality to detect an identified sensor output identifying the gesture; and

storing the identified sensor output in the memory further comprises;

storing in the memory the identified sensor outputs only of the at least sensor associated with at least one selected sensor modality.

16. A tangible computer-readable memory device storing instructions that, when executed on a processor of a computing device having a sensor, cause the computing device to perform actions associated with gestures performed by a user, by:

transitioning to a training mode upon receiving. from the user, a training start request;

during the training mode;

while the user performs the gesture, monitoring the sensor to detect an identified sensor output identifying the gesture during a training period between the training start request and the training completion request;

identifying, during the training, period, a restricted period that is shorter than the training period and during which the identified sensor output identifies the gesture;

associating with the gesture the identified sensor output of the sensor during the restricted period; and

associating an action with the gesture;

transitioning to a recognition mode upon receiving, from the user, a training completion request; and

during the recognition mode;

monitoring the sensor output of the sensor; and

upon detecting that the sensor output matches the identified sensor output associated with the gesture, performing the action associated with the gesture.

17. The tangible computer-readable memory device of claim 16, wherein executing the instructions on the processor further causes the computing device to:

store an association of the gesture with the action and the identified sensor output of the sensor in a gesture profile; and

send the gesture profile to a gesture profile service.

18. The tangible computer-readable memory device of claim 16, wherein executing the instructions on the processor further causes the computing device to retrieve a gesture profile from a gesture profile service, the gesture profile comprising at least one association of a gesture with an identified sensor output of a sensor and an action.

19. The tangible computer-readable memory device of claim 16, wherein:

the action is further associated with a device context of the computing device; and

performing the action further comprises;

upon detecting that the sensor output matches the identified sensor output associated with the gesture while a current device context of the computing device matches the device context associated with the action, performing the action associated with the gesture.

20. The tangible computer-readable memory device of claim 16, wherein:

the computing device comprises at least two sensors;

the action is further associated with the sensor providing the identified sensor output that is associated with the gesture; and

performing the action further comprises;

upon detecting that the sensor output matches the identified sensor output associated with the gesture, performing the action associated with the gesture and with the sensor providing the sensor output.