Systems and methods for classifying and representing gestural inputs

• US 7,783,492 B2
• Filed: 07/24/2008
• Issued: 08/24/2010
• Est. Priority Date: 08/17/2001
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Johnston, Michael J., Bangalore, Srinivas
• Primary Examiner(s): Han; Qi
• Application Number:
  US12/179,223
• Publication Number:
  US 20080306737A1
• Time in Patent Office: 761 Days
• Field of Search: 704/275, 704/270, 704/270.1, 704/231, 704/235
• US Class Current: 704/275
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 8/35 model driven
• CPC Class Codes: G06F 8/37 Compiler construction; Parser generation
• CPC Class Codes: G06V 30/1985 Syntactic analysis, e.g. using a grammatical approach syntactic image representation G06V30/18171
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

Gesture and handwriting recognition agents provide possible interpretations of electronic ink. Recognition is performed on both individual strokes and combinations of strokes in the input ink lattice. The interpretations of electronic ink are classified and encoded as symbol complexes where symbols convey specific attributes of the contents of the stroke. The use of symbol complexes to represent strokes in the input ink lattice facilitates reference to sets of entities of a specific type.

Claims

1. A method for classifying and representing information, comprising:

recognizing an utterance containing information;

converting the information into at least one classification scheme, each classification scheme comprising a sequence of symbols including a SEM placeholder symbol for specific content of the utterance, wherein each symbol identifies an attribute of the information contained in the utterance, and wherein the specific content is mapped to the SEM placeholder symbol and other specific attributes are mapped to themselves; and

outputting the classification scheme.

2. The method of claim 1, further comprising generating the utterance using at least two of a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem and a body motion input subsystem.

3. The method of claim 1, wherein the attribute identified by the symbol is at least one of form, meaning and specific content of the utterance containing information.

4. The method of claim 3, wherein the form attribute identifies the physical form of the utterance containing information.

5. The method of claim 4, wherein the meaning attribute identifies the meaning of the form attribute.

6. The method of claim 4, wherein the form attribute has a value of at least one of area, line and arrow.

7. The method of claim 6, wherein the form attribute has a value of area and the meaning attribute identifies a meaning of area.

8. A system for classifying and representing information, comprising:

a processor;

a module configured to control the processor to recognize an utterance containing information;

a module configured to control the processor to convert the information into at least one classification scheme, each classification scheme comprising a sequence of symbols including a SEM placeholder symbol for specific content of the utterance, wherein each symbol identifies an attribute of the information contained in the utterance, and wherein the specific content is mapped to the SEM placeholder symbol and other specific attributes are mapped to themselves; and

a module configured to control the processor to output the classification scheme.

9. The system of claim 8, further comprising a module configured to control the processor to generate the utterance using two or more of a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem and a body motion input subsystem.

10. The system of claim 8, wherein the attribute identified by the symbol is at least one of form, meaning and specific content of the utterance containing information.

11. The system of claim 10, wherein the form attribute identifies the physical form of the utterance containing information.

12. The system of claim 11, wherein the meaning attribute identifies the meaning of the form attribute.

13. The system of claim 11, wherein the form attribute has a value of at least one of area, line and arrow.

14. The system of claim 13, wherein the form attribute has a value of area and the meaning attribute identifies a meaning of area.

15. A tangible computer-readable medium storing instructions for controlling a computing device to classify and represent information, the instructions comprising:

recognizing an utterance containing information;

converting the information into at least one classification scheme, each classification scheme comprising a sequence of symbols including a SEM placeholder symbol for specific content of the utterance, wherein each symbol identifies an attribute of the information contained in the utterance, and wherein the specific content is mapped to the SEM placeholder symbol and other specific attributes are mapped to themselves; and

outputting the classification scheme.

16. The computer-readable medium of claim 15, wherein the instructions further comprise generating the utterance using two or more of a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem and a body motion input subsystem.

17. The tangible computer-readable medium of claim 15, wherein the attribute identified by the symbol is at least one of form, meaning and specific content of the utterance containing information.

18. The tangible computer-readable medium of claim 17, wherein the form attribute identifies the physical form of the utterance containing information.

19. The tangible computer-readable medium of claim 18, wherein the meaning attribute identifies the meaning of the form attribute.

20. The tangible computer-readable medium of claim 18, wherein the form attribute has a value of at least one of area, line and arrow.

21. The tangible computer-readable medium of claim 20, wherein the form attribute has a value of area and the meaning attribute identifies a meaning of area.