Systems and methods for classifying and representing gestural inputs

• US 7,505,908 B2
• Filed: 08/12/2002
• Issued: 03/17/2009
• Est. Priority Date: 08/17/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Bangalore, Srinivas, Johnston, Michael J.
• Primary Examiner(s): Han; Qi
• Application Number:
  US10/216,390
• Publication Number:
  US 20030046087A1
• Time in Patent Office: 2,409 Days
• Field of Search: 704/275, 704/270, 704/3, 704/270.1
• US Class Current: 704/275
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 8/35 model driven
• CPC Class Codes: G06F 8/37 Compiler construction; Parser generation
• CPC Class Codes: G06V 30/1985 Syntactic analysis, e.g. using a grammatical approach syntactic image representation G06V30/18171
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

Gesture and handwriting recognition agents provide possible interpretations of electronic ink. Recognition is performed on both individual strokes and combinations of strokes in the input ink lattice. The interpretations of electronic ink are classified and encoded as symbol complexes where symbols convey specific attributes of the contents of the stroke. The use of symbol complexes to represent strokes in the input ink lattice facilitates reference to sets of entities of a specific type.

Claims

1. A method for representing gestures made by a user, comprising:

recognizing a gesture;

converting the recognized gesture into at least one recognition result by using finite-state transducers, each recognition result comprising a sequence of symbols including a set of gesture symbols with a SEM (semantics) placeholder symbol for specific content of the gesture, wherein each symbol of the sequence of symbols identifies a specific attribute of the recognized gesture and the SEM placeholder symbol represents the specific content of the gesture such that when recomposing specific gestural content, the specific content is mapped to the SEM placeholder symbol and other specific attributes are mapped to themselves; and

outputting the recognition result.

2. The method of claim 1, further comprising generating the gesture using at least one of a pen input subsystem, a computer vision input subsystem, a haptic subsystem, a gaze input subsystem, and a body motion input subsystem.

3. The method of claim 1, wherein the attribute identified by the symbol is at least one of FORM, MEANING and specific content of the recognized gesture, the specific content being represented by the SEM placeholder symbol in the sequence of symbols.

4. The method of claim 3, wherein the FORM attribute identifies the physical form of the recognized gesture.

5. The method of claim 3, wherein the MEANING attribute identifies the meaning of the FORM attribute.

6. The method of claim 5, wherein the FORM attribute has a value of at least one of area, line and arrow.

7. The method of claim 6, wherein the FORM attribute has a value of area and the MEANING attribute identifies a meaning of area.

8. The method of claim 3, wherein the MEANING attribute has a value of at least one of location and selection.

9. The method of claim 8, wherein, when the MEANING attribute has a value of selection, the recognition result uses a symbol for at least one of a number of entities selected by the gesture and a type of the entities selected by the gesture.

10. The method of claim 9, wherein the TYPE attribute identifies a type of entity or entities selected.

11. The method of claim 10, wherein the TYPE attribute has a value of at least one of at least restaurant, theatre, hotel and mixed.

12. The method of claim 3, wherein when the attribute identified by the symbol is specific content, the recognition result comprises at least one of entity identifiers and points selected by a user.

13. A system for recognizing strokes made on an input device, comprising:

a recognition device usable to recognize gestures; and

a converting device that converts the recognized gesture into at least one recognition result by using finite-state transducers, each recognition result comprising a sequence of symbols including a set of gesture symbols with a SEM (semantics) placeholder symbol for specific content of the gesture, wherein each symbol of the sequence of symbols identifies a specific attribute of the recognized gesture and the SEM placeholder symbol represents the specific content of the gesture such that when recomposing specific gestural content, the specific content is mapped to the SEM placeholder symbol and other specific attributes are mapped to themselves; and

an output device that outputs the recognition result.

14. The system of claim 13, wherein the gesture is made using at least one of a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem and a body motion input subsystem.

15. The system of claim 13, wherein the attribute identified by the symbol is at least one of FORM, MEANING and specific content of the recognized gesture.

16. The system of claim 15, wherein the FORM attribute identifies the physical form of the recognized gesture.

17. The system of claim 16, wherein the FORM attribute has a value of at least one of area, line and arrow.

18. The system of claim 17, wherein the FORM attribute has a value of area and the MEANING attribute identifies the meaning of area.

19. The system of claim 15, wherein the MEANING attribute identifies the meaning of the FORM attribute.

20. The system of claim 15, wherein the MEANING attribute has a value of at least one of location and selection.

21. The system of claim 20, wherein when the MEANING attribute has a value of selection, the recognition result further comprises a symbol for at least one of a number of entities selected by the gesture and a type based on at least one of the entities selected by the gesture.

22. The system of claim 21, wherein the TYPE attribute identifies the type of entity or entities selected.

23. The system of claim 21, wherein the TYPE attribute has a value of at least one of restaurant, theatre, hotel and mixed.

24. The method of claim 13, wherein, when the attribute identified by the symbol is specific content, the recognition result comprises at least one of entity identifiers and points selected by a user.

25. A method for free-form gesture representation, comprising:

recognizing a gesture;

converting the recognized gesture into at least one recognition result by using finite-state transducers, each recognition result comprising a sequence of symbols including a set of gesture symbols with a SEM (semantics) placeholder symbol for specific content of the gesture, wherein each symbol of the sequence of symbols identifies a specific attribute of the recognized gesture and the SEM placeholder symbol represents the specific content of the gesture such that when recomposing specific gestural content, the specific content is mapped to the SEM placeholder symbol and other specific attributes are mapped to themselves;

generating a recognition lattice based on the recognition result; and

outputting the recognition lattice.

26. The method of claim 25, further comprising generating the gesture using at least one of a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem, and a body motion input subsystem.

27. The method of claim 25, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the symbols to identify at least one attribute identified by the symbol including at least one of FORM, MEANING and specific content of the recognized gesture.

28. The method of claim 27, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using a FORM attribute to identify the physical form of the recognized gesture.

29. The method of claim 28, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using a MEANING attribute to identify the meaning of the FORM attribute.

30. The method of claim 29, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the FORM attribute having at least a value of at least one of area, line and arrow.

31. The method of claim 25, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the symbols to identify at least one attribute identified by the symbol including at least one of FORM, MEANING and specific content of the recognized gesture, wherein a FORM attribute has a value of area and a MEANING attribute identifies the meaning of area.

32. The method of claim 25, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the symbols to identify at least one attribute identified by the symbol including at least one of FORM, MEANING and specific content of the recognized gesture, wherein a meaning attribute has a value of at least one of location and selection.

33. The method of claim 25, comprising converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the symbols to identify at least one attribute identified by the symbol including at least one of FORM, MEANING and specific content of the recognized gesture, wherein, when a MEANING attribute has a value of selection, the recognition result further comprises a symbol for at least one of a number of entities selected by the gesture and a type based on at least one of the entities selected by the gesture.

34. The method of claim 33, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using a TYPE attribute to identify the type of each of at least one entity selected.

35. The method of claim 34, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the TYPE attribute to identify the type of each of at least one entity selected, wherein the TYPE attribute has a value of at least one of at least restaurant, theatre, hotel and mixed.

36. The method of claim 25, wherein converting the recognized gesture into at least one recognition result having a sequence of symbols comprises using the symbols to identify specific content including at least one of entity identifiers and points selected by a user.