Systems and methods for abstracting portions of information that is represented with finite-state devices

• US 20030065505A1
• Filed: 08/12/2002
• Published: 04/03/2003
• Est. Priority Date: 08/17/2001
• Status: Abandoned Application

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Bangalore, Srinivas, Johnston, Michael J.
• Application Number:
  US10/216,392
• Publication Number:
  US 20030065505A1
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 8/35 model driven
• CPC Class Codes: G06F 8/37 Compiler construction; Parser generation
• CPC Class Codes: G06V 30/1985 Syntactic analysis, e.g. using a grammatical approach syntactic image representation G06V30/18171
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

When using finite-state methods to capture multimodal integration, it is desirable to abstract over certain aspects of the gestural content. For abstraction, the gestural input is represented as a transducer which relates the specific content of the gestures to the gesture symbols. After multimodal integration, a projection is taken of the resulting finite state machine and it is composed with the transducer representing the gestural input in order to reincorporate the specific contents that were left out of the finite-state process.

Claims

1. A method of abstracting over certain components of information contained in a finite-state device, comprising:

representing information using a first finite-state transducer that relates a first set of symbols representing the information to a second set of symbols representing the information, wherein the first set of symbols includes at least a first subset of symbols representing certain components of the information and the second set of symbols includes at least a second subset of symbols, different from the first subset of symbols, representing the certain components of the information;

using a projection of the second set of symbols to perform a function using finite-state devices;

generating a result finite-state machine that relates at least the second set of symbols representing the information and at least one other symbol representing other information to a set of symbols representing an output of the performed function;

taking a projection of the result finite-state machine, wherein the projection contains the second set of symbols and the set of symbols representing the output of the performed function;

composing the projection of the result finite-state machine with the first finite-state transducer; and

generating a final finite-state machine that relates the first set of symbols to the symbols representing the output of the performed function, wherein a result of the performed function may be obtained by concatenating the symbols representing the output of the performed function and when the symbol is one of the symbols from the second subset of symbols, inserting a related symbol from the first subset of symbols.

2. The method of claim 1, wherein:

the first finite-state transducer comprises a plurality of transitions between states, each transition having at least one input symbol and at least one output symbol; and

the at least one input symbol is at least one of the first subset of symbols and the at least one output symbol is at least one of the second subset of symbols.

3. The method of claim 1, wherein representing information using a first finite-state transducer comprises receiving information input by a user in an input portion of a device.

4. The method of claim 3, wherein representing certain components of the information includes representing at least one of at least one entity identifier and at least one point on the input portion that were selected by the user.

5. The method of claim 3, wherein:

the result finite-state machine comprises a plurality of transitions between states, each transition having at least one input symbol and at least one output symbol, wherein the at least one input symbol is at least one of the second subset of symbols;

the at least one output symbol is at least one of a third set of symbols representing the result of the performed function;

a result of the performed function is obtained by concatenating the third set of symbols in the result finite-state machine; and

replacing the symbol from the third set of symbols with the related input symbol of the result finite-state machine when the symbol of the third set of symbols is a symbol from a subset of the second set of symbols representing certain components of the information.

6. The method of claim 1, wherein the symbols, in any of the first-fourth sets of symbols representing the information, represent attributes of the information and the attributes identified by the symbols are at least one of form, meaning and specific content of the information represented by the first finite-state transducer.

7. The method of claim 1, further comprising inputting the information using at least one of a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem, and a body motion input subsystem.

8. A method for recognizing a multimodal utterance comprising a plurality of different modes, comprising:

inputting at least a first mode of the multimodal utterance and a second-mode of the multimodal utterance that is different than the first mode;

generating a first mode recognition finite state transducer that relates a first set of symbols representing the first mode of the multimodal utterance to a second set of symbols representing the first mode of the multimodal utterance, wherein the symbols in the second set of symbols representing specific contents of the first mode are different from the symbols in the first set of symbols representing the specific contents of the first mode;

generating a first mode recognition lattice from the first mode recognition finite-state transducer, wherein the first mode recognition lattice contains the second set of symbols;

performing a function using the first mode recognition lattice;

generating a result finite-state device that at least relates symbols from the second set of symbols to a third set of symbols representing the output of the performed function;

taking a projection of the result finite-state device, wherein the projection contains symbols from the second set of symbols representing the first mode of the multimodal input and symbols from the third set of symbols representing the output of the performed function;

composing the projection of the result finite-state device with the first mode recognition finite-state transducer; and

generating a second finite-state machine that relates the first set of symbols representing the first mode of the multimodal utterance and specific contents of the first mode of the multimodal utterance to a fourth set of symbols, wherein;

a result of the performed function can be obtained by concatenating the fourth set of symbols representing the output of the performed function, and

when the symbol in the fourth set of symbols is one of the symbols from the second set of symbols representing the specific contents of the first mode, replacing the symbol from the fourth set of symbols with the symbol from the first set of symbols representing the specific contents of the first mode.

9. The method of claim 8, wherein the symbols, in any of the first-fourth sets of symbols representing the first mode of the multimodal utterance, represent attributes of the first mode of the multimodal utterance and the attributes identified by the first set of symbols are at least one of form, meaning and specific content of the first mode of the multimodal utterance.

10. The method of claim 9, wherein the specific contents of the first mode of the multimodal utterance includes at least one of at least one entity identifier and at least one point, on a first mode input portion, that were selected by an user.

11. The method of claim 10, wherein the second set of symbols representing the first mode of the multimodal utterance include at least one symbol representing the specific contents of the first mode of the multimodal utterance.

12. The method of claim 11, wherein:

the function performed using the first mode recognition lattice is multimodal integration; and

the method further comprises;

generating a possible meaning of the multimodal utterance by concatenating the fourth set of symbols representing the multimodal recognition, and

obtaining and inserting, when a symbol in the fourth set of symbols is a symbol representing the specific contents of first mode of the multimodal recognition, the symbol from the first set of symbols representing the specific contents of the first mode of the multimodal utterance in place of the symbol in the fourth set of symbols representing the specific contents of the first mode of the multimodal utterance.

13. A method for abstracting over specifics of multimodal input content, comprising:

representing a first mode of a multimodal input as a first mode finite-state transducer relating a first set of symbols representing the first mode of the multimodal input to a second set of symbols representing the first mode of the multimodal input, wherein the symbols in the first set of symbols representing specific contents of the first mode are different from the symbols in the second set of symbols representing the specific contents of the first mode;

taking a projection of the output of the first mode finite-state transducer;

using the projection of the output of the first mode finite-state transducer to carry out multimodal integration;

generating a meaning finite state machine, which relates the first mode of the multimodal input and at least one other recognized input mode with meaning by performing multimodal integration;

taking a projection of the meaning finite-state machine, wherein the projection contains a third set of symbols representing the first mode of the multimodal input and a set of symbols representing a meaning of the multimodal input;

composing the projection of the meaning finite-state machine with the first mode finite-state transducer; and

generating a first-mode-input-meaning finite-state machine that relates the first set of symbols representing the first mode of the multimodal input and specific contents of the first mode of the multimodal input to a fourth set of symbols representing the multimodal meaning of the multimodal input, wherein;

a result of the multimodal integration can be obtained by concatenating the fourth set of symbols, and

when the symbol in the fourth set of symbols represents the specific contents of the first mode, replacing the symbol with the related symbol from the first set of symbols.

14. The method of claim 13, wherein the specific contents of the first mode of the multimodal input includes at least one of at least one entity identifier and at least one point, on a first mode input portion, that were selected by an user.

15. The method of claim 14, further comprising:

generating a possible meaning of the multimodal input by concatenating the fourth set of symbols representing the multimodal meaning, wherein when a symbol in the fourth set of symbols is a symbol representing the specific contents of first mode of the multimodal input; and

obtaining and inserting the symbol in the first set of symbols representing the specific contents of the first mode of the multimodal input in place of the symbol in the fourth set of symbols representing the specific contents of the first mode of the multimodal input.

16. A method for abstracting over specifics of gestural content, comprising:

converting a gesture recognition lattice, representing an input gesture, to a gesture finite-state transducer that relates a first set of gesture symbols representing the input gesture to a second set of symbols representing the input gesture, wherein the symbols representing specific contents of the input gesture are different in the first and second set of symbols;

taking a projection of an output of the gesture-input finite-state transducer;

using the projection of the output of the gesture-input finite-state transducer to perform a function and to generate a result finite-state machine;

taking a projection of an output of the result finite-state machine, wherein the projection of the finite state machine includes a third set of symbols;

composing the projection of the output of the result finite-state machine with the gesture-input finite state transducer;

generating an input gesture-result finite state machine that relates the first set of gesture symbols to the third set of symbols representing the output of the performed function, wherein a result of the output can be obtained by concatenating the symbols in the third set of symbols and, when the symbol in the third set of symbols is a symbol representing specific contents of the gesture, replacing the symbol with the symbol from the first set of symbols representing the specific contents of the gesture.

17. The method of claim 16, further comprising converting the gesture recognition lattice, representing the input gesture, to a gesture finite-state transducer, wherein the specific contents of the input gesture includes at least one of entity identifiers and points, on a first mode input portion, that were selected by an user.

18. The method of claim 16, further comprising generating the input gesture.

19. The method of claim 18, wherein generating the input gesture comprises selecting at least one of at least one entity identifier and at least one point, on a first mode input portion.

20. A method for abstracting over specifics of gestural content, comprising:

representing a gestural input using a gesture-input finite-state transducer that relates a first set of gesture symbols representing a gestural input to a second set of gesture symbols representing the gestural input, wherein specific contents of the gestural input are represented by different symbols in the first and second set of gesture symbols;

taking a projection of an output of the gesture-input finite-state transducer;

performing multimodal integration using the projection of the output of the gesture input finite-state transducer;

generating a meaning finite-state machine that relates the input gesture and at least one other recognized input mode with meaning;

generating a gesture-meaning finite-state machine from the meaning finite-state machine by taking a projection of the meaning finite-state machine, wherein the projection contains the input gesture and the meaning of the meaning finite-state machine;

composing the gesture-meaning finite-state machine with the gesture-input finite-state transducer; and

generating an input-meaning finite-state machine, wherein a meaning of the.

21. The method of claim 20, further comprising representing the gestural input using a gesture-input finite-state transducer, wherein:

the gesture-input finite-state transducer comprises a plurality of transitions between states, each transition having at least one input symbol and at least one output symbol;

the at least one input symbol is at least one of a first set of gesture symbols representing the gestural input and specific contents of the gestural input;

the at least one output symbol is at least one of a second set of gesture symbols representing attributes of the gestural input and at least one symbol representing the specific contents of the gestural input contained on the input path; and

specific contents of the gesture include at least one of at least one entity identifier and at least one point, on a gesture input portion, that were selected by an user.

22. The method of claim 21, further comprising generating an input-meaning finite-state machine, comprising:

creating a plurality of transitions between states;

selecting, for each transition, at least one input symbol and at least one output symbol, wherein;

selecting the at least one input symbol comprises selecting at least one of the first set of gesture symbols representing the gestural input and specific contents of the gestural input, and

selecting as the at least one output symbol comprises selecting at least one of a third set of symbols representing the meaning of the multimodal input;

obtaining a meaning of the multimodal input by concatenating the third set of symbols in the input-meaning finite-state machine; and

acquiring, when the symbol of the third set of symbols is a symbol representing the specific contents of the gestural input, the specific contents of the gesture input from the input symbol of the input-meaning finite-state machine.

23. The method of claim 20, further comprising representing the input gesture with symbols representing attributes of the gestural input, wherein the symbols represent attributes of the gestural input including at least one of at least one form, at least one meaning and at least one specific content of the input gesture.

24. The method of claim 20, further comprising generating the gesture input using at least one of a pen input subsystem, a computer vision subsystem, a haptic subsystem, a gaze subsystem, and a body motion subsystem.