Dynamic feedback for gestures

• US 7,886,236 B2
• Filed: 03/28/2003
• Issued: 02/08/2011
• Est. Priority Date: 03/28/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Kolmykov-Zotov, Alexander J, Lerner, Matt, Somji, Shiraz
• Primary Examiner(s): Pitaro; Ryan F
• Application Number:
  US10/401,903
• Publication Number:
  US 20050275638A1
• Time in Patent Office: 2,874 Days
• Field of Search: 715856-864
• US Class Current: 715/863
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Gesture feedback techniques are discussed that provide prompt feedback to a user concerning the recognition of one or more gestures. The feedback may be employed to confirm to a user that a gesture is being correctly recognized. The feedback may alternately warn a user that a desired gesture is not being correctly recognized, thereby allowing the user to cancel the erroneous gesture before it is invoked

Claims

1. A method of providing dynamic feedback to a user making a handwriting gesture, the method comprising:

(i) receiving position data from a pointing device, the position data being produced by the use of the pointing device;

(ii) analyzing the received position data to recognize when the received position data corresponds to a handwriting gesture command;

(iii) if the received position data corresponds to at least one handwriting gesture command, determining that the received position data represents the at least one handwriting gesture command, providing first feedback indicating to a user recognition of the at least one handwriting gesture command,

receiving additional position data that does not correspond to the at least one handwriting gesture command, wherein the additional position data and the received position data are produced by a continuous use of the pointing device and wherein the receipt of the additional position data indicates that the recognition of the at least one handwriting gesture is not desired by the user,

canceling the determination that the received position data represents the at least one handwriting gesture command; and

providing feedback indicating the cancellation;

(iv) if the received position data does not correspond to at least one handwriting gesture command, determining that the received position data does not represent a handwriting gesture command, and automatically providing second feedback indicating to a user that a handwriting gesture command has not been recognized, wherein the second feedback includes passive feedback;

(v) repeating steps (i) through (iv) until no further position data is being received; and

(vi) when no further position data is being received, if the received position data has been determined to be representing a single handwriting gesture command, invoking the single handwriting gesture command.

2. The method recited in claim 1, further comprising:

recognizing that initially received position data corresponds to a plurality of handwriting gesture commands; and

upon recognizing that the initially received position data corresponds to plurality of handwriting gesture commands, providing the first feedback so as to indicate recognition of each of the plurality of handwriting gesture commands.

3. The method recited in claim 2, further comprising:

determining a confidence in an accuracy of the recognition of each of the plurality of handwriting gesture commands, and providing the first feedback so as to indicate recognition of the plurality of handwriting gesture commands based upon the determined confidence.

4. The method recited in claim 3, further comprising providing the first feedback by visually rendering an indicator identifying each of the plurality of handwriting gesture commands, such that an appearance of each indicator corresponds to the confidence determined for the accuracy of the recognition of the handwriting gesture command identified by the indicator.

5. The method recited in claim 3, further comprising:

analyzing subsequently received position data to determine a second confidence in the accuracy of the recognition of each of the plurality of the handwriting gesture commands; and

modifying the second feedback based upon the determined second confidence.

6. The method recited in claim 2, further comprising:

analyzing subsequently received position data to recognize a single handwriting gesture command corresponding to the received position data; and

providing third feedback indicating to a user recognition of the single handwriting gesture command.

7. The method recited in claim 6, further comprising providing the third feedback so as to delete any indication of recognition of each of the plurality of handwriting gesture commands other than the single handwriting gesture command.

8. The method recited in claim 1, further comprising:

recognizing that initially received position data corresponds to a plurality of handwriting gesture commands determining a confidence in an accuracy of the recognition of each of the plurality of handwriting gesture commands such that at least one of the plurality of handwriting gesture commands has a highest determined confidence, and providing the first feedback so as to indicate recognition of only the at least one handwriting gesture command having the highest determined confidence.

9. The method recited in claim 1, further comprising visually providing the feedback to a user, wherein a form of the feedback and a display position of the feedback is based upon the location of the position data.

10. The method recited in claim 9, further comprising visually providing the feedback to the user by including the feedback in at least one transparent window rendered on a display.

11. The method recited in claim 10, further comprising visually providing the feedback to the user by including the feedback in a plurality of transparent windows rendered on a display.

12. The method recited in claim 1, further comprising audibly providing the feedback to a user.

13. The method recited in claim 1, further comprising:

visually rendering location information corresponding to the position data on a display; and

visually rendering at least the first feedback on the display.

14. The method recited in claim 13, further comprising rendering at least the first feedback at a position on the display relative to a location at which the location information is rendered on the display.

15. The method recited in claim 14, further comprising moving the position at which at least first feedback is rendered on the display relative to changes in the rendering of the location information on the display.

16. The method recited in claim 14, further comprising rendering at least the first feedback at a position on the display relative to a location at which the location information is rendered on the display based upon preference information provided by the user.

17. The method recited in claim 13, further comprising rendering both the location information and at least the first feedback in a character input user interface.

18. The method recited in claim 17, wherein the character input user interface includes a soft Qwerty keyboard, and the first feedback includes a change in appearance of a soft key of the soft Qwerty keyboard corresponding to the at least one handwriting gesture command.

19. The method recited in claim 1, wherein the position data is produced only in response to movement of the pointing device.

20. The method recited in claim 1, wherein the second feedback is a deletion of the first feedback.

21. The method recited in claim 1, further comprising providing the second feedback so as to indicate that the at least one handwriting gesture command is no longer being recognized.

22. A method of providing dynamic feedback to a user making a handwriting gesture, the method comprising:

receiving a first portion of a position data stream from a pointing device, the position data stream being produced by the use of the pointing device to make a handwriting gesture;

recognizing a plurality of handwriting gesture commands corresponding to the received first portion of the position data stream;

upon recognizing that the first portion of the position data stream corresponds to a plurality of handwriting gesture commands, providing feedback indicating recognition of each of the plurality of handwriting gesture commands;

determining a degree of confidence in an accuracy of the recognition of each of the plurality of handwriting gesture commands corresponding to the received first portion of the position data stream,

before receiving a second portion of the position data stream, providing feedback to a user indicating recognition of the plurality of handwriting gesture commands, wherein the feedback includes a perceptible indication as to the degree of determined confidence in the accuracy of the recognition of each of the plurality of handwriting gesture commands;

receiving additional position data that does not correspond to the plurality of handwriting gesture commands, wherein the additional position data and the received position data are produced by a continuous use of the pointing device and wherein the receipt of the additional position data indicates that the recognition of the plurality of handwriting gesture commands is not desired by the user;

canceling the recognition of the plurality of handwriting gesture commands; and

providing feedback indicating the cancellation.

23. The method recited in claim 22, further comprising, receiving the second portion of the position data stream;

and when the position data stream is no longer being received, invoking a handwriting gesture command from among the plurality of handwriting gesture commands recognized from the received position data stream.

24. The method recited in claim 22, further comprising visually rendering an indicator identifying each of the plurality of handwriting gesture commands, such that an appearance of each indicator corresponds to the confidence determined for the accuracy of the recognition of the handwriting gesture command identified by the indicator.

25. The method recited in claim 24, further comprising:

receiving the second portion of the position data stream;

using the received second portion of the position data stream to determine a second confidence in the accuracy of the recognition of each of the plurality of the handwriting gesture commands; and

providing second feedback indicating recognition of the plurality of handwriting gesture commands based upon the determined second confidence.

26. The method recited in claim 25, further comprising:

designating that a single handwriting gesture command among the plurality of handwriting gesture commands has a highest second determined confidence, and providing the second feedback by deleting any indication of recognition of each of the plurality of handwriting gesture commands other than the single handwriting gesture command.

27. The method recited in claim 22, further comprising:

determining a confidence in an accuracy of the recognition of each of the plurality of handwriting gesture commands such that at least one of the plurality of handwriting gesture commands has a highest determined confidence, and providing the feedback by indicating recognition of only the at least one handwriting gesture command having the highest determined confidence.

28. The method recited in claim 22, further comprising:

after providing the feedback to the user, receiving the second portion of the position data stream from the pointing device;

analyzing the second portion of the position data stream to recognize a single handwriting gesture command corresponding to the received first and second portions of position data stream from among the plurality of handwriting gesture commands; and

providing second feedback indicating to a user recognition of the single handwriting gesture command.

29. The method recited in claim 28, further comprising:

receiving a third portion of the position data stream;

analyzing the third portion of the position data stream to recognize if the third portion of the position data stream corresponds to at least one of the plurality of handwriting gesture commands; and

if the third portion of the position data stream does not correspond to at least one of the plurality of handwriting gesture commands, providing third feedback indicating that a handwriting gesture command is no longer being recognized.

30. The method recited in claim 29, further comprising providing the third feedback by deleting the second feedback.

31. The method recited in claim 29, further comprising providing the third feedback by generating an indicator indicating that the single handwriting gesture command is no longer being recognized.

32. The method recited in claim 22, further comprising visually providing the feedback to a user, wherein a form of the feedback and a display position of the feedback is based upon the location of the position data.

33. The method recited in claim 32, further comprising visually providing the feedback to the user by including the feedback in at least one transparent window rendered on a display.

34. The method recited in claim 33, further comprising visually providing the feedback to the user by including the feedback in a plurality of transparent windows rendered on a display.

35. The method recited in claim 22, further comprising audibly providing the feedback to a user.

36. The method recited in claim 22, further comprising:

visually rendering location information corresponding to the position data on a display; and

visually rendering the feedback on the display.

37. The method recited in claim 36, further comprising rendering the feedback at a position on the display relative to a location at which the location information is rendered on the display.

38. The method recited in claim 37, further comprising moving the position at which the feedback is rendered on the display relative to changes in the rendering of the location information on the display.

39. The method recited in claim 37, further comprising rendering the feedback at a position on the display relative to a location at which the location information is rendered on the display based upon preference information provided by the user.

40. The method recited in claim 36, further comprising rendering both the location information and the feedback in a character input user interface.

41. The method recited in claim 40, wherein the character input user interface includes a soft Qwerty keyboard corresponding to the at least one handwriting gesture command, and the feedback includes a change in appearance of a soft key of the Qwerty keyboard corresponding to the at least one handwriting gesture command.

42. The method recited in claim 22, wherein the position data is produced only in response to movement of the pointing device.

43. The method recited in claim 22, further including adjusting the feedback responsive to the determined confidence, wherein a degree of confidence is reflected in the feedback.

44. The method recited in claim 22, further comprising:

receiving a second portion of the position data stream;

analyzing the second portion of the position data stream to recognize if the second portion of the position data stream corresponds to the at least one handwriting gesture command; and

if the second portion of the position data stream does not correspond to the at least one handwriting gesture command, providing second feedback indicating that a handwriting gesture command is no longer being recognized.

45. The method recited in claim 44, further comprising providing the second feedback by deleting the first feedback.

46. The method recited in claim 44, further comprising providing the second feedback by generating an indicator indicating that a handwriting gesture command is no longer being recognized.

47. A gesture feedback tool, embodied in computer storage media, for providing dynamic feedback to a user making a handwriting gesture, the gesture feedback tool comprising:

a gesture recognition module that;

receives a first portion of a position data stream and a second portion of the position data stream from a pointing device, wherein the first and second portions of the position data stream are produced by use of the pointing device;

continuously analyzes the received position data stream to recognize when the received position data stream corresponds to a plurality of handwriting gesture commands; and

in response to recognizing that the first portion of the position data stream corresponds to the plurality of handwriting gesture commands before receiving the second portion of the position data stream, produces position data identifying the plurality of handwriting gesture commands corresponding to the received first portion of the position data stream;

a gesture confidence module that;

determines a confidence in an accuracy of the recognition of each of the plurality of handwriting gesture commands corresponding to the received first portion of the position data stream; and

a gesture feedback module that;

receives the position data from the gesture recognition module; and

in response to receiving the position data and based upon the determined confidence, produces feedback for a user corresponding to the plurality of handwriting gesture commands identified by the position data corresponding to the received first portion of the position data stream, wherein the feedback includes a perceptible indication as to a degree of confidence attributable to each of the plurality of handwriting gesture commands;

receives additional position data that does not correspond to the plurality of handwriting gesture commands, wherein the additional position data is received via an uninterrupted use of the pointing device and wherein the receipt of the additional position data indicates that the recognition of the plurality of handwriting gesture commands is not desired by the user;

cancels the recognition of the plurality of handwriting gesture commands;

provides feedback indicating that the plurality of handwriting gesture commands are no longer recognized.

48. The gesture feedback tool recited in claim 47, wherein the gesture recognition module includes a neural net.

49. The gesture feedback tool recited in claim 47, wherein the gesture recognition module includes a recognizer for recognizing that the received position data corresponds to at least one handwriting gesture command based upon heuristics.

50. The gesture feedback tool recited in claim 49, wherein the gesture recognition module includes a neural net.

51. The gesture feedback tool recited in claim 47, wherein the received position data is associated with pointing locations on a display;

and the gesture feedback module includes a position calculation module that visually renders the feedback on the display relative to the pointing locations.

52. The feedback tool recited in claim 51, wherein the position calculation module visually renders the feedback on the display relative to the pointing locations based upon the location of the position data.

53. The feedback tool recited in claim 47, wherein the gesture feedback module includes a layered window management module that renders the feedback in a user interface.

54. The feedback tool recited in claim 53, wherein the layered window management module renders the feedback in a character input user interface.

55. The feedback tool recited in claim 53, wherein the layered window management module renders the feedback in one or more transparent windows.

56. The feedback tool recited in claim 47, wherein the feedback reflects the confidence in the accuracy of the recognition of each of the plurality of handwriting gesture commands.