Dynamic feedback for gestures

• US 20050275638A1
• Filed: 03/28/2003
• Published: 12/15/2005
• Est. Priority Date: 03/28/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lerner, Matt, Somji, Shiraz, Kolmykov-Zotov, Alexander J.
• Granted Patent: US 7,886,236 B2
• Time in Patent Office: Days
• US Class Current: 345/179
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Gesture feedback techniques are discussed that provide prompt feedback to a user concerning the recognition of one or more gestures. The feedback may be employed to confirm to a user that a gesture is being correctly recognized. The feedback may alternately warn a user that a desired gesture is not being correctly recognized, thereby allowing the user to cancel the erroneous gesture before it is invoked

Claims

1. A method of providing dynamic feedback to a user making a gesture, comprising:

receiving position data from a pointing device, the position data being produced by the use of the pointing device to make a gesture;

continuously analyzing the received position data to recognize when the received data corresponds to at least one gesture command; and

upon recognizing that the analyzed position data corresponds to at least one gesture command, providing feedback indicating to a user recognition of the at least one gesture command.

2. The method recited in claim 1, further comprising:

subsequent to providing the feedback, continuing to receive the position data from the pointing device, and

when the subsequently received position data is no longer being received, invoking a gesture command from among the at least one gesture command recognized from the received position data.

3. The method recited in claim 1, further comprising recognizing that the analyzed position data corresponds to plurality of gesture commands.

4. The method recited in claim 3, further comprising, upon recognizing that the analyzed position data corresponds to plurality of gesture commands, providing feedback indicating recognition of each of the plurality of gesture commands.

5. The method recited in claim 4, further comprising:

determining a confidence in an accuracy of the recognition of each of the plurality of gestures commands, and

providing the feedback indicating recognition of the plurality of gesture commands based upon the determined confidence.

6. The method recited in claim 5, further comprising visually rendering an indicator identifying each of the plurality of gesture commands, such that an appearance of each indicator corresponds to the confidence determined for the accuracy of the recognition of the gesture command identified by the indicator.

7. The method recited in claim 5, further comprising:

subsequent to providing the feedback, continuing to receive the position data from the pointing device, and

using the subsequently received position data to determine a second confidence in the accuracy of the recognition of each of the plurality of the gesture commands; and

providing second feedback indicating recognition of the plurality of gesture commands based upon the determined second confidence.

8. The method recited in claim 4, further comprising:

subsequent to providing the feedback, continuing to receive the position data from the pointing device,

analyzing the subsequently received position data to recognize a single gesture command corresponding to the received position data from among the plurality of gesture commands; and

providing feedback indicating to a user recognition of the single gesture command.

10. The method recited in claim 3, further comprising

determining a confidence in an accuracy of the recognition of each of the plurality of gesture commands such that at least one of the plurality of gesture commands has a highest determined confidence, and

providing feedback indicating recognition of only the at least one gesture command having the highest determined confidence.

11. The method recited in claim 1, further comprising visually providing the feedback to a user.

12. The method recited in claim 11, further comprising visually providing the feedback to the user by including the feedback in at least one transparent window rendered on a display.

13. The method recited in claim 12, further comprising visually providing the feedback to the user by including the feedback in a plurality of transparent windows rendered on a display.

14. The method recited in claim 1, further comprising audibly providing the feedback to a user.

15. The method recited in claim 1, further comprising:

visually rendering location information corresponding to the position data on a display; and

visually rendering the feedback on the display.

16. The method recited in claim 15, further comprising rendering the feedback at a position on the display relative to a location at which the location information is rendered on the display.

17. The method recited in claim 16, further comprising moving the position at which the feedback is rendered on the display relative to changes in the rendering of the location information on the display.

18. The method recited in claim 16, further comprising rendering the feedback at a position on the display relative to a location at which the location information is rendered on the display based upon preference information provided by the user.

19. The method recited in claim 15, further comprising rendering both the location information and the feedback in a character input user interface.

20. The method recited in claim 19, wherein

the character input user interface includes a soft key corresponding to the at least one gesture command, and

the feedback includes a change in appearance of the soft key corresponding to the at least one gesture command.

21. The method recited in claim 1, wherein the position data is produced only in response to movement of the pointing device.

22. The method recited in claim 1, further comprising:

subsequent to providing the feedback, continuing to receive the position data from the pointing device,

continuously analyzing the subsequently received position data to recognize when the subsequently received data corresponds to the at least one gesture command; and

upon recognizing that the analyzed position data does not correspond to the at least one gesture command, providing second feedback indicating to a user that the at least one gesture command is no longer being recognized.

23. The method recited in claim 22, further comprising providing the second feedback by deleting the first feedback.

24. The method recited in claim 22, further comprising providing the providing the second feedback by generating an indicator indicating that the at least one gesture command is no longer being recognized.

25. The method recited in claim 22, further comprising:

subsequent to providing the second feedback, continuing to receive the position data from the pointing device, and

when the position data is no longer being received, not invoking a gesture command.

9. The method recited in 8, further comprising, upon analyzing the subsequently received position data to recognize a single gesture command corresponding to the received position data, deleting the feedback indicating recognition of each of the plurality of gesture commands other than the single gesture command.

26. A method of providing dynamic feedback to a user making a gesture, comprising:

(i) receiving position data from a pointing device, the position data being produced by the use of the pointing device;

(ii) analyzing the received position data to recognize when the received position data corresponds to a gesture command;

(iii) if the received position data corresponds to at least one gesture command,

determining that the received position data represents the at least one gesture command is being, and

providing first feedback indicating to a user recognition of the at least one gesture command;

(iv) if the received position data does not correspond to at least one gesture command,

determining that the received position data does not represent a gesture command, and

providing second feedback indicating to a user that a gesture command has not been recognized;

(v) repeating steps (i) through (iv) until no further position data is being received; and

(vi) when no further position data is being received, if the received position data has been determined to be representing a single gesture command, invoking the single gesture command.

27. The method recited in claim 26, further comprising:

recognizing that initially received position data corresponds to plurality of gesture commands; and

upon recognizing that the initially received position data corresponds to plurality of gesture commands, providing the first feedback so as to indicate recognition of each of the plurality of gesture commands.

28. The method recited in claim 27, further comprising:

determining a confidence in an accuracy of the recognition of each of the plurality of gestures commands, and

providing the first feedback so as to indicate recognition of the plurality of gesture commands based upon the determined confidence.

29. The method recited in claim 28, further comprising providing the first feedback by visually rendering an indicator identifying each of the plurality of gesture commands, such that an appearance of each indicator corresponds to the confidence determined for the accuracy of the recognition of the gesture command identified by the indicator.

30. The method recited in claim 28, further comprising:

analyzing subsequently received position data to determine a second confidence in the accuracy of the recognition of each of the plurality of the gesture commands; and

modifying the second feedback based upon the determined second confidence.

33. The method recited in claim 26, further comprising:

recognizing that initially received position data corresponds to plurality of gesture commands

determining a confidence in an accuracy of the recognition of each of the plurality of gesture commands such that at least one of the plurality of gesture commands has a highest determined confidence, and

providing the first feedback so as to indicate recognition of only the at least one gesture command having the highest determined confidence.

34. The method recited in claim 26, further comprising visually providing the feedback to a user.

35. The method recited in claim 34, further comprising visually providing the feedback to the user by including the feedback in at least one transparent window rendered on a display.

36. The method recited in claim 35, further comprising visually providing the feedback to the user by including the feedback in a plurality of transparent windows rendered on a display.

37. The method recited in claim 26, further comprising audibly providing the feedback to a user.

38. The method recited in claim 26, further comprising:

visually rendering location information corresponding to the position data on a display; and

visually rendering at least the first feedback on the display.

39. The method recited in claim 38, further comprising rendering at least the first feedback at a position on the display relative to a location at which the location information is rendered on the display.

40. The method recited in claim 39, further comprising moving the position at which at least first feedback is rendered on the display relative to changes in the rendering of the location information on the display.

41. The method recited in claim 39, further comprising rendering at least the first feedback at a position on the display relative to a location at which the location information is rendered on the display based upon preference information provided by the user.

42. The method recited in claim 38, further comprising rendering both the location information and at least the first feedback in a character input user interface.

43. The method recited in claim 42, wherein

the character input user interface includes a soft key corresponding to the at least one gesture command, and

the first feedback includes a change in appearance of the soft key corresponding to the at least one gesture command.

44. The method recited in claim 26, wherein the position data is produced only in response to movement of the pointing device.

45. The method recited in claim 26, wherein the second feedback is a deletion of the first feedback.

46. The method recited in claim 26, further comprising providing the second feedback so as to indicate that the at least one gesture command is no longer being recognized.

31. The method recited in 27, further comprising:

analyzing subsequently received position data to recognize a single gesture command corresponding to the received position data; and

providing third feedback indicating to a user recognition of the single gesture command.

32. The method recited in 31, further comprising providing the third feedback so as to delete any indication of recognition of each of the plurality of gesture commands other than the single gesture command.

47. A method of providing dynamic feedback to a user making a gesture, comprising:

receiving a first portion of a position data stream from a pointing device, the position data stream being produced by the use of the pointing device to make a gesture;

recognizing at least one gesture command corresponding to the received first portion of the position data stream; and

before receiving a second portion of the position data stream, providing feedback to a user indicating recognition of the at least one gesture command.

48. The method recited in claim 47, further comprising, when the position data stream is no longer being received, invoking a gesture command from among the at least one gesture command recognized from the received position data stream.

49. The method recited in claim 47, further comprising:

recognizing that the first portion of the position data stream corresponds to a plurality of gesture commands.

50. The method recited in claim 49, further comprising, upon recognizing that the first portion of the position data stream corresponds to a plurality of gesture commands, providing feedback indicating recognition of each of the plurality of gesture commands.

51. The method recited in claim 50, further comprising:

determining a confidence in an accuracy of the recognition of each of the plurality of gestures commands, and

providing the feedback indicating recognition of the plurality of gesture commands based upon the determined confidence.

52. The method recited in claim 51, further comprising visually rendering an indicator identifying each of the plurality of gesture commands, such that an appearance of each indicator corresponds to the confidence determined for the accuracy of the recognition of the gesture command identified by the indicator.

53. The method recited in claim 52, further comprising:

receiving the second portion of the position data stream;

using the received second portion of the position data stream to determine a second confidence in the accuracy of the recognition of each of the plurality of the gesture commands; and

providing second feedback indicating recognition of the plurality of gesture commands based upon the determined second confidence.

55. The method recited in claim 49, further comprising

determining a confidence in an accuracy of the recognition of each of the plurality of gesture commands such that at least one of the plurality of gesture commands has a highest determined confidence, and

providing the feedback by indicating recognition of only the at least one gesture command having the highest determined confidence.

56. The method recited in claim 49, further comprising:

after providing the feedback to the user, receiving the second portion of the position data stream from the pointing device;

analyzing the second portion of the position data stream to recognize a single gesture command corresponding to the received first and second portions of position data stream from among the plurality of gesture commands; and

providing second feedback indicating to a user recognition of the single gesture command.

57. The method recited in claim 56, further comprising:

receiving a third portion of the position data stream;

analyzing the third portion of the position data stream to recognize if the third portion of the position data stream corresponds to at least one of the plurality of gesture commands; and

if the third portion of the position data stream does not correspond to at least one of the plurality of gesture commands, providing third feedback indicating that a gesture command is no longer being recognized.

58. The method recited in claim 57, further comprising providing the third feedback by deleting the second feedback.

59. The method recited in claim 57, further comprising providing the third feedback by generating an indicator indicating that the single gesture command is no longer being recognized.

60. The method recited in claim 47, further comprising visually providing the feedback to a user.

61. The method recited in claim 60, further comprising visually providing the feedback to the user by including the feedback in at least one transparent window rendered on a display.

62. The method recited in claim 61, further comprising visually providing the feedback to the user by including the feedback in a plurality of transparent windows rendered on a display.

63. The method recited in claim 47, further comprising audibly providing the feedback to a user.

64. The method recited in claim 47, further comprising:

visually rendering location information corresponding to the position data on a display; and

visually rendering the feedback on the display.

65. The method recited in claim 64, further comprising rendering the feedback at a position on the display relative to a location at which the location information is rendered on the display.

66. The method recited in claim 65, further comprising moving the position at which the feedback is rendered on the display relative to changes in the rendering of the location information on the display.

67. The method recited in claim 65, further comprising rendering the feedback at a position on the display relative to a location at which the location information is rendered on the display based upon preference information provided by the user.

68. The method recited in claim 64, further comprising rendering both the location information and the feedback in a character input user interface.

69. The method recited in claim 68, wherein

the character input user interface includes a soft key corresponding to the at least one gesture command, and

the feedback includes a change in appearance of the soft key corresponding to the at least one gesture command.

70. The method recited in claim 47, wherein the position data is produced only in response to movement of the pointing device.

71. The method recited in claim 47, further comprising:

receiving a second portion of the position data stream;

analyzing the second portion of the position data stream to recognize if the second portion of the position data stream corresponds to the at least one gesture command; and

if the second portion of the position data stream does not correspond to the at least one gesture command, providing second feedback indicating that a gesture command is no longer being recognized.

72. The method recited in claim 71, further comprising providing the second feedback by deleting the first feedback.

73. The method recited in claim 71, further comprising providing the second feedback by generating an indicator indicating that a gesture command is no longer being recognized.

54. The method recited in 53, further comprising:

designating that a single gesture command among the plurality of gesture commands has a highest second determined confidence, and

providing the second feedback by deleting any indication of recognition of each of the plurality of gesture commands other than the single gesture command.

74. A gesture feedback tool for providing dynamic feedback to a user making a gesture, comprising:

a gesture recognition module that

receives pointing data from a pointing device;

continuously analyzes the received pointing data to recognize when the received pointing data corresponds to at least one gesture command; and

in response to recognizing that the received pointing data corresponds to at least one gesture command, produces position data identifying the at least one gesture command;

a gesture feedback module that

receives the gesture command data from the gesture recognition module, and

in response to receiving the gesture command data, produces feedback for a user corresponding to the at least one gesture identified by the gesture command data.

75. The gesture feedback tool recited in claim 74, wherein the gesture recognition module includes a neural net.

76. The gesture feedback tool recited in claim 74, wherein the gesture recognition module includes a recognizer for recognizing that the received pointing data corresponds to at least one gesture command based upon heuristics.

77. The gesture feedback tool recited in claim 76, wherein the gesture recognition module includes a neural net.

78. The gesture feedback tool recited in claim 74, wherein

the received pointing data is associated with pointing locations on a display;

and

the gesture feedback module includes a position calculation module that visually renders the feedback on the display relative to the pointing locations.

79. The feedback tool recited in claim 78, wherein the position calculation module visually renders the feedback on the display relative to the pointing locations based upon preference information provided by a user.

80. The feedback tool recited in claim 74, wherein the gesture feedback module includes a layered window management module that renders the feedback in a user interface.

81. The feedback tool recited in claim 80, wherein the layered window management module renders the feedback in a character input user interface.

82. The feedback tool recited in claim 80, wherein the layered window management module renders the feedback in one or more transparent windows.

83. A method of recognizing a gesture using feedback, comprising:

providing feedback to a user corresponding to at least one gesture command;

receiving data produced by the user associated with the feedback; and

recognizing at least one gesture command from the association of the received data with the feedback.

84. The method recited in claim 83, wherein the feedback is an indication of movement of a pointing device corresponding to the at least one gesture command.

85. The method recited in claim 83, wherein the feedback is an indication of a location through which a pointer should be moved to make a gesture.

86. The method recited in claim 83, wherein the feedback is an indication of a location at which a pointer should begin moving to make a gesture.