Gesture recognition interactive feedback

• US 7,770,136 B2
• Filed: 01/24/2007
• Issued: 08/03/2010
• Est. Priority Date: 01/24/2007
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kennedy, Kevin A., Bilow, Charles R., Beeck, Langdon W.
• Primary Examiner(s): Bashore; William L
• Assistant Examiner(s): UM, DANIEL H
• Application Number:
  US11/626,794
• Publication Number:
  US 20080178126A1
• Time in Patent Office: 1,287 Days
• Field of Search: 715/863
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G09B 19/003 Repetitive work cycles; Sequence of movements
• CPC Class Codes: G09B 5/06 with both visual and audible presentation of the material to be studied

Abstract

Example embodiments provide for (i) suggesting proper gesture sequences based on a user'"'"'s initial gesture input; and/or (ii) providing visual or other gesture example(s), which may alternatively be compared with the user'"'"'s mimic thereof. For example, a user may input a gesture into a touch/display surface. Based on the user'"'"'s movement, recognized or proper application gesture commands are suggested. In other embodiments, a tutorial session may be presented to the user for teaching or educating the user on numerous types of gestures. As such, a visual representation of a gesture example may be presented to the user, and feedback provided based on the user'"'"'s mimicking thereof.

Claims

1. At a gesture recognition tutorial system including a motion-sensitive display surface capable of recognizing various forms of user interaction with the surface, a method of suggesting recognized application gesture commands based on the user interaction with the surface in order to give the user various options and/or instructions on how to appropriately use the application for the motion-sensitive display surface, the method comprising:

receiving at a motion-sensitive display surface a user-performed gesture, which includes user movement of an object over the surface that recognizes such user interaction therewith;

determining that the received user-performed gesture is an unrecognized command, the gesture not corresponding to gesture commands recognized by one or more programs running on the gesture recognition tutorial system'"'"'s motion-sensitive display surface;

based on the determination, identifying one or more gesture commands that are similar to the received user-performed gesture that are recognizable by the one or more programs of the motion-sensitive display surface, wherein the one or more gesture commands recognized by the program represent actual commands of the program; and

displaying one or more exemplary gesture commands recognized by the one or more programs in order to aid the user in understanding appropriate various defined gestures representing actual program commands, the exemplary gesture commands comprising one or more graphical, non-textual images that depict a series of one or more object motions or movements performable by the user that form a recognized command, recognizable by the motion-sensitive display surface.

2. The method of claim 1, wherein the user movement includes one or more of tapping, touching, touching and holding for a specified time, sliding, waving, flicking or dragging.

3. The method of claim 1, wherein the object is one or more of a bodily appendage, writing utensil, stylus, dice, paint brushes, cellular telephones, coins, poker chips, or radio frequency identification (RFID) tagged object.

4. The method of claim 1, wherein the user-performed gesture closely resembles one or more gestures commands for one or more programs.

5. The method of claim 4, wherein a pre-defined degree of variance is used to identify the gesture command based on the difference between the user-performed gesture and a recognized gesture.

6. The method of claim 1, further comprising interacting with the user to determine whether the displayed gesture was the gesture intended by the user'"'"'s movement input.

7. The method of claim 1, wherein the displayed exemplary gesture commands comprise textual information.

8. The method of claim 1, wherein the displayed exemplary gesture commands comprise pictorial information.

9. The method of claim 1, wherein the displayed exemplary gesture commands are superimposed over a visual depiction of the user-performed gesture.

10. The method of claim 1, wherein the displayed exemplary gesture commands comprise a visual deformation of the motion-sensitive display surface.

11. The method of claim 1, wherein the displayed exemplary gesture commands are displayed in real-time, as the user is inputting gestures.

12. The method of claim 1, further comprising allowing the user to create a gesture that is recognized by one or more programs as a program command.

13. The method of claim 1, wherein the gesture recognition tutorial system processes all possible inputs within the context of the detected movement inputs while the gesture is being performed.

14. The method of claim 13, further comprising dynamically displaying the possible interpretations of the user'"'"'s gestures inputs in real-time.

15. At a gesture recognition tutorial system including a motion-sensitive display surface capable of recognizing various forms of user interaction with the surface simultaneously from a plurality users, a method of providing a gesture tutorial mechanism for instructing one or more users how to properly enter program commands via the motion-sensitive display surface using gestures, the method comprising:

determining that a received user-performed gesture is an unrecognized command, the gesture not corresponding to gesture commands recognized by one or more programs running on the gesture recognition tutorial system'"'"'s motion-sensitive display surface;

identifying one or more gesture commands similar to the received user-performed gesture that are recognized by a program of a motion-sensitive display surface, the motion-sensitive display surface being configured to detect multiple simultaneous user movements of objects over the surface from a plurality of simultaneous motion-sensitive display surface users, the surface being configured to recognize such multi-user interaction therewith; and

based on the identified one or more gesture commands, displaying a visual example of a gesture capable of being performed on the motion-sensitive display surface, wherein the visual example of the gesture comprises one or more graphical, non-textual images that depict one or more motions a user can perform for executing the one or more gesture commands of a program running on the motion-sensitive display surface.

16. The method of claim, 15 further comprising:

at the motion-sensitive display surface, detecting a user-performed gesture;

based on the user-performed gesture, determining if the user-performed gesture appropriately corresponded to the example gesture; and

based upon the determination, indicating whether or not the user-performed gesture was proper.

17. The method of claim 15, wherein the visual example gesture corresponds to a command generally recognized by the gesture recognition tutorial system.

18. The method of claim 15, wherein the gesture recognition tutorial system processes all possible inputs within the context of the detected user movements while the gesture is being performed.

19. The method of claim 15, further comprising labeling the visual example with a label that indicates the gesture command displayed in the visual example.

20. At a gesture recognition tutorial system including a motion-sensitive display surface, a computer program product comprising one or more recordable-type computer-readable storage media having thereon computer-executable instructions that, when executed by one or more processors of a computer system, cause the computer system to perform a method for suggesting recognized application gesture commands based on the user interaction with the surface, the method comprising:

receiving at a motion-sensitive display surface a user-performed gesture, which includes user movement of an object over the surface that recognizes such user interaction therewith;

determining that the received user-performed gesture is an unrecognized command, the gesture not corresponding to gesture commands recognized by one or more programs running on the gesture recognition tutorial system'"'"'s motion-sensitive display surface;

based on the determination, identifying one or more gesture commands that are similar to the received user-performed gesture that are recognizable by the one or more programs running on the gesture recognition tutorial system'"'"'s motion-sensitive display surface, the identified commands representing actual commands of the program; and

displaying one or more exemplary gesture commands recognized by the one or more programs in order to aid the user in understanding appropriate various defined gestures representing actual program commands, the exemplary gesture commands comprising one or more graphical, non-textual images that depict a series of one or more object motions or movements performable by the user that form a recognized command, recognizable by the motion-sensitive display surface.