Customized gesture interpretation

• US 10,620,709 B2
• Filed: 01/15/2014
• Issued: 04/14/2020
• Est. Priority Date: 04/05/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Ultrahaptics Ip Two Limited (Ultraleap Holdings Ltd.)
• Inventors: Holz, David
• Primary Examiner(s): Nguyen, Tuan S
• Application Number:
  US14/156,429
• Publication Number:
  US 20140304665A1
• Time in Patent Office: 2,281 Days
• Field of Search: None
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

The technology disclosed relates to filtering gestures, according to one implementation. In particular, it relates to distinguishing between interesting gestures from non-interesting gestures in a three-dimensional (3D) sensory space by comparing characteristics of user-defined reference gestures against characteristics of actual gestures performed in the 3D sensory space. Based on the comparison, a set of gestures of interest are filtered from all the gestures performed in the 3D sensory space. The technology disclosed also relates to customizing gesture interpretation for a particular user, according to another implementation. In particular, it relates to setting parameters for recognizing gestures by prompting the user to select values for characteristics of the gestures. In one implementation, the technology disclosed includes performing characteristic focused demonstrations of boundaries of the gesture. It further includes testing the interpretation of gestures by prompting the user to perform complete gesture demonstrations and receiving evaluation from the user regarding the interpretation.

Claims

1. A method of a gesture system performing user-customized gesture recognition in a three-dimensional (3D) sensory space, the method including:

in a user-initiated training environment, training the gesture system to recognize a user-defined customized gesture by;

detecting, by an electronic sensor, a training gesture performed by the user in the 3D sensory space;

determining a set of characteristic values for the detected training gesture using data captured by the electronic sensor; and

storing the set of characteristic values for recognizing the training gesture as the user-defined customized gesture; and

in a gesture interaction environment, performing gesture recognition of the user-defined customized gesture by;

detecting an actual gesture performed by the user in the 3D sensory space using an electronic sensor;

determining a set of characteristic values for the detected actual gesture using data from the electronic sensor;

matching the determined set of characteristic values for the detected actual gesture with the stored set of characteristic values for the training gesture from among multiple stored sets of characteristic values;

identifying the training gesture, for which the set of characteristic values matched the set of characteristic values of the detected actual gesture, as a gesture of interest; and

displaying an on-screen indicator to reflect a degree of completion of the gesture of interest as the detected actual gesture is being performed by the user in real time, thereby enabling the user to dynamically control a relationship between actual movement of the user in real time and a corresponding action displayed in real time on the screen,

wherein the degree of completion is displayed by the on-screen indicator in real time as the user continues to perform the detected actual gesture to reflect the degree of completion of the actual gesture, in real time, as performed by the user while dynamically interacting with virtual objects displayed on the screen.

2. The method of claim 1, further including identifying a set of gestures of interest by matching the determined set of characteristic values for the detected actual gesture with sets of characteristic values for multiple training gestures and providing the set of gestures of interest to a further process.

3. The method of claim 2, further including interpreting lateral sweeps as at least included in the set of gestures of interest, and wherein providing the set of gestures of interest further includes providing at least actual gestures with lateral sweeps to the further process.

4. The method of claim 2, further including a reference characteristic that includes a gesture velocity, and the method further including interpreting actual gestures with high velocities as at least included in the set of gestures of interest, and wherein providing the set of gestures of interest further includes providing at least actual gestures with a reference characteristic that indicates a high gesture velocity to the further process.

5. The method of claim 2, further including a reference characteristic that includes a gesture configuration, and the method further including interpreting actual gestures made using a hand pointing with a particular finger as at least included in the set of gestures of interest, and wherein providing the set of gestures of interest further includes providing at least actual gestures with a reference characteristic that indicates a gesture made using a hand pointing with a particular finger to the further process.

6. The method of claim 2, further including a reference characteristic that includes a gesture configuration, and the method further including interpreting actual gestures of hands making a first as at least included in the set of gestures of interest, and wherein providing the set of gestures of interest further includes providing at least actual gestures with a reference characteristic that indicates a gesture made using a hand making a first to the further process.

7. The method of claim 2, further including a reference characteristic that includes a gesture shape, and the method further including interpreting actual gestures of hands with thumbs-up as at least included in the set of gestures of interest, and wherein providing the set of gestures of interest further includes providing at least actual gestures with a reference characteristic that indicates a gesture made using a hand with a thumbs-up to the further process.

8. The method of claim 2, further including a reference characteristic that includes a gesture length, and the method further including interpreting waving gestures as at least included in the set of gestures of interest, and wherein providing the set of gestures of interest further includes providing at least actual gestures with a reference characteristic that indicates a waving gesture to the further process.

9. The method of claim 2, further including a reference characteristic that includes a gesture position, and the method further including interpreting actual gestures made by a hand within a threshold proximity to the electronic sensor as at least included in the set of gestures of interest, and wherein providing the set of gestures of interest further includes providing at least actual gestures with a reference characteristic that indicates a gesture made within a threshold proximity to the electronic sensor to the further process.

10. The method of claim 2, further including a reference characteristic that includes gesture duration, further including interpreting actual gestures within the 3D sensory space for a threshold time-period as at least included in the set of gestures of interest rather than actual gestures within the 3D sensory space for time-period less than the threshold time-period, and wherein providing the set of gestures of interest further includes providing at least actual gestures with a reference characteristic that indicates a gesture made within a threshold time-period to the further process.

11. The method of claim 1, further including:

prompting a user to respond to questions about characteristics of performing a first gesture including information about a minimum and maximum amount of time the first gesture takes to perform, threshold time-period, number of hands and fingers involved in making the first gesture, and whether size, speed, or direction of the first gesture matters;

responsive to receiving a response from the user that size of the first gesture matters, prompting the user to demonstrate the smallest and largest reasonable movements to perform the first gesture in a three-dimensional (3D) sensory space;

determining, from demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor, a set of movement parameter values of the first gesture; and

storing the set of movement parameter values of the first gesture for recognizing subsequent instances of the first gesture performed by the user.

12. The method of claim 11, further including using a questionnaire for prompting the user to select values for characteristics of the gesture.

13. The method of claim 12, wherein prompting the user to select values for characteristics of the gesture using the questionnaire includes receiving from the user a minimum threshold time-period for the gesture to be within the 3D sensory space before which the gesture is not interpreted.

14. The method of claim 11, wherein demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor includes the user making a hand pointing gesture with a particular finger as a gesture configuration.

15. The method of claim 11, wherein demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor includes the user making a first gesture with a hand as a gesture configuration.

16. The method of claim 11, wherein demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor includes the user making a thumbs-up gesture with a hand as a gesture shape.

17. The method of claim 11, wherein demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor includes the user making a thumb-down gesture with a hand as a gesture shape.

18. The method of claim 11, wherein demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor includes the user making a pinch gesture to set a minimum gesture distance as a gesture size.

19. The method of claim 11, wherein demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor includes the user making a waving gesture to set a maximum gesture distance as a gesture size.

20. The method of claim 11, wherein demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor includes the user making a finger-flicking gesture to set a fastest gesture motion.

21. The method of claim 11, wherein demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor includes the user making a waving gesture to set a slowest gesture motion.

22. The method of claim 11, wherein demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor includes the user making a lateral sweep to set a straight gesture path.

23. The method of claim 11, demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor includes the user making a circular sweep to set a circular gesture path.

24. The method of claim 11, further including testing interpretation of a particular gesture by:

prompting the user to perform a complete gesture demonstration of the particular gesture in the 3D sensory space;

determining, from the complete gesture demonstration captured by the electronic sensor, a set of parameters of the particular gesture;

comparing the set of parameters of the particular gesture with corresponding set of parameters determined from the user demonstration of the smallest and largest reasonable movements; and

reporting results of the comparing to the user and receiving a confirmation on whether interpretation of the particular gesture is correct.

25. The method of claim 11, wherein the user is a first user, and further comprising:

prompting a second user to respond to questions about performing the first gesture;

responsive to the second user demonstrating smallest and largest reasonable movements to perform the first gesture in a three-dimensional (3D) sensory space, determining a set of movement parameters values of the gesture; and

storing the set of movement parameters values for recognizing the first gesture performed by the second user.

26. The method of claim 25, further comprising:

detecting one or more actual gestures performed by the second user in a three-dimensional (3D) sensory space using an electronic sensor and determining actual characteristic values for the one or more actual gestures detected using data from the electronic sensor;

retrieving the characteristic values for recognizing the first gesture performed by the second user stored;

recognizing a set of gestures of interest by matching the actual characteristic values of the one or more actual gestures with the characteristic values retrieved; and

providing the set of gestures of interest recognized to a further process.

27. The method of claim 11, wherein a movement parameter is one of curvature of path, velocity, direction, and attributes of an object making the gesture.

28. The method of claim 11, wherein a gesture characteristic identified for assigning a user-customized value is selected by the user.

29. The method of claim 11, wherein a gesture characteristic identified for assigning a user-customized value is selected by a gesture training system that leads the user through a sequence of assigning characteristic values.

30. The method of claim 1, further including storing a set of object parameters defining at least one object displayable on a contact-free display and associated with the gesture.

31. A non-transitory computer readable medium storing computer program instructions to customize gesture interpretation for a particular user, the instructions, when executed on a processor, implement a method including:

in a user-initiated training environment, training a gesture system to recognize a user-defined customized gesture by;

detecting, by an electronic sensor, a training gesture performed by the user in a three-dimensional (3D) sensory space;

determining a set of characteristic values for the detected training gesture using data captured by the electronic sensor; and

storing the set of characteristic values for recognizing the training gesture as the user-defined customized gesture; and

in a gesture interaction environment, performing gesture recognition of the user-defined customized gesture by;

detecting an actual gesture performed by the user in a the 3D sensory space using an electronic sensor;

determining a set of characteristic values for the detected actual gesture, using data from the electronic sensor;

matching the determined set of characteristic values for the detected actual gesture with the stored set of characteristic values for the training gesture from among multiple stored sets of characteristic values;

identifying the training gesture, for which the set of characteristic values matched the set of characteristic values of the detected actual gesture, as a gesture of interest; and

displaying an on-screen indicator to reflect a degree of completion of the gesture of interest as the detected actual gesture is being performed by the user in real time, thereby enabling the user to dynamically control a relationship between actual movement of the user in real time and a corresponding action displayed in real time on the screen,

wherein the degree of completion is displayed by the on-screen indicator in real time as the user continues to perform the detected actual gesture to reflect the degree of completion of the actual gesture, in real time, as performed by the user while dynamically interacting with virtual objects displayed on the screen.

32. The non-transitory computer readable medium of claim 31, the method further including:

prompting a user to respond to questions about characteristics of performing a first gesture including information about a minimum and maximum amount of time the first gesture takes to perform, threshold time-period, number of hands and fingers involved in making the first gesture, and whether size, speed, or direction of the first gesture matters;

responsive to receiving a response from the user that size of the first gesture matters, prompting the user to demonstrate the smallest and largest reasonable movements to perform the first gesture in a three-dimensional (3D) sensory space;

determining, from demonstration by the user of the smallest and largest reasonable movements of the user captured by an electronic sensor, a set of movement parameter values of the first gesture; and

storing the set of movement parameter values of the first gesture for recognizing subsequent instances of the first gesture performed by the user.