Systems and methods for classifying and representing gestural inputs

• US 20030046087A1
• Filed: 08/12/2002
• Published: 03/06/2003
• Est. Priority Date: 08/17/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Bangalore, Srinivas, Johnston, Michael J.
• Granted Patent: US 7,505,908 B2
• Time in Patent Office: Days
• US Class Current: 704/275
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 8/35 model driven
• CPC Class Codes: G06F 8/37 Compiler construction; Parser generation
• CPC Class Codes: G06V 30/1985 Syntactic analysis, e.g. using a grammatical approach syntactic image representation G06V30/18171
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

Gesture and handwriting recognition agents provide possible interpretations of electronic ink. Recognition is performed on both individual strokes and combinations of strokes in the input ink lattice. The interpretations of electronic ink are classified and encoded as symbol complexes where symbols convey specific attributes of the contents of the stroke. The use of symbol complexes to represent strokes in the input ink lattice facilitates reference to sets of entities of a specific type.

Claims

1. A method for representing gestures made by a user, comprising:

recognizing a gesture;

converting the recognized gesture into at least one recognition result, each recognition result comprising a sequence of symbols, wherein each symbol identifies an attribute of the recognized gesture; and

outputting the recognition result.

2. The method of claim 1, further comprising generating the gesture using at least one of a pen input subsystem, a computer vision input subsystem, a haptic subsystem, a gaze input subsystem, and a body motion input subsystem.

3. The method of claim 1, wherein the attribute identified by the symbol is at least one of form, meaning and specific content of the recognized gesture.

4. The method of claim 3, wherein the form attribute identifies the physical form of the recognized gesture.

5. The method of claim 3, wherein the meaning attribute identifies the meaning of the form attribute.

6. The method of claim 5, wherein the form attribute has a value of at least one of area, line and arrow.

7. The method of claim 6, wherein the form attribute has a value of area and the meaning attribute identifies a meaning of area.

8. The method of claim 3, wherein the meaning attribute has a value of at least one of location and selection.

9. The method of claim 8, wherein, when the meaning attribute has a value of selection, the recognition result uses a symbol for at least one of a number of entities selected by the gesture and a type of the entities selected by the gesture.

10. The method of claim 9, wherein the type attribute identifies a type of entity or entities selected.

11. The method of claim 10, wherein the type attribute has a value of at least one of at least restaurant, theatre, hotel and mixed.

12. The method of claim 3, wherein when the attribute identified by the symbol is specific content, the recognition result comprises at least one of entity identifiers and points selected by a user.

13. A method for classifying and representing information, comprising:

recognizing an utterance containing information;

converting the information into at least one classification scheme, each classification scheme comprising a sequence of symbols, wherein each symbol identifies an attribute of the information contained in the utterance; and

outputting the classification scheme.

14. The method of claim 13, further comprising generating the utterance using a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem and a body motion input subsystem.

15. The method of claim 13, wherein the attribute identified by the symbol is at least one of form, meaning and specific content of the utterance containing information.

16. The method of claim 15, wherein the form attribute identifies the physical form of the utterance containing information.

17. The method of claim 16, wherein the meaning attribute identifies the meaning of the form attribute.

18. The method of claim 16, wherein the form attribute has a value of at least one of area, line and arrow.

19. The method of claim 18, wherein the form attribute has a value of area and the meaning attribute identifies a meaning of area.

20. A system for recognizing strokes made on an input device, comprising:

a recognition device usable to recognize gestures; and

a converting device that converts the recognized gesture into at least one recognition result, the recognition result comprising a sequence of symbols, wherein each symbol identifies an attribute of the recognized gesture.

21. The system of claim 20, wherein the gesture is made using at least one of a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem and a body motion input subsystem.

22. The system of claim 20, wherein the attribute identified by the symbol is at least one of form, meaning and specific content of the recognized gesture.

23. The system of claim 22, wherein the form attribute identifies the physical form of the recognized gesture.

24. The system of claim 22, wherein the meaning attribute identifies the meaning of the form attribute.

25. The system of claim 23, wherein the form attribute has a value of at least one of area, line and arrow.

26. The system of claim 25, wherein the form attribute has a value of area and the meaning identifies the meaning of area.

27. The system of claim 22, wherein the meaning attribute has a value of at least one of location and selection.

28. The system of claim 27, wherein when the meaning attribute has a value of selection, the recognition result further comprises a symbol for at least one of a number of entities selected by the gesture and a type based on at least one of the entities selected by the gesture.

29. The system of claim 28, wherein the type attribute identifies the type of entity or entities selected.

30. The system of claim 28, wherein the type attribute has a value of at least one of restaurant, theatre, hotel and mixed.

31. The method of claim 20, wherein, when the attribute identified by the symbol is specific content, the recognition result comprises at least one of entity identifiers and points selected by a user.

32. A method for free-form gesture representation, comprising:

recognizing a gesture;

converting the recognized gesture into at least one recognition result comprising a sequence of symbols, wherein each symbol identifies an attribute of the recognized gesture;

generating a recognition lattice based on the recognition result; and

outputting the recognition lattice.

33. The method of claim 32, further comprising generating the gesture using at least one of a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem, and a body motion input subsystem.

34. The method of claim 32, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the symbols to identify at least one attribute identified by the symbol including at least one of form, meaning and specific content of the recognized gesture.

35. The method of claim 34, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using a form attribute to identify the physical form of the recognized gesture.

36. The method of claim 35, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using a meaning attribute to identify the meaning of the form attribute.

37. The method of claim 36, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the form attribute having at least a value of at least one of area, line and arrow.

38. The method of claim 32, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the symbols to identify at least one attribute identified by the symbol including at least one of form, meaning and specific content of the recognized gesture, wherein a form attribute has a value of area and a meaning attribute identifies the meaning of area.

39. The method of claim 32, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the symbols to identify at least one attribute identified by the symbol including at least one of form, meaning and specific content of the recognized gesture, wherein a meaning attribute has a value of at least one of location and selection.

40. The method of claim 32, comprising converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the symbols to identify at least one attribute identified by the symbol including at least one of form, meaning and specific content of the recognized gesture, wherein, when a meaning attribute has a value of selection, the recognition result further comprises a symbol for at least one of a number of entities selected by the gesture and a type based on at least one of the entities selected by the gesture.

41. The method of claim 40, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using a type attribute to identify the type of each of at least one entity selected.

42. The method of claim 41, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the type attribute to identify the type of each of at least one entity selected, wherein the type attribute has a value of at least one of at least restaurant, theatre, hotel and mixed.

43. The method of claim 32, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the symbols to identify specific content including at least one of entity identifiers and points selected by a user.