Systems and methods for generating markup-language based expressions from multi-modal and unimodal inputs

• US 8,433,731 B2
• Filed: 12/22/2009
• Issued: 04/30/2013
• Est. Priority Date: 10/24/2002
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Johnston, Michael J., Bangalore, Sirnivas
• Primary Examiner(s): Choi, Yuk Ting
• Application Number:
  US12/644,603
• Publication Number:
  US 20100100509A1
• Time in Patent Office: 1,225 Days
• Field of Search: 707/803, 707/796
• US Class Current: 707/803
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/151 Transformation
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

When using finite-state devices to perform various functions, it is beneficial to use finite state devices representing regular grammars with terminals having markup-language-based semantics. By using markup-language-based symbols in the finite state devices, it is possible to generate valid markup-language expressions by concatenating the symbols representing the result of the performed function. The markup-language expression can be used by other applications and/or devices. Finite-state devices are used to convert strings of words and gestures into valid markup-language, for example, XML, expressions that can be used, for example, to provide an application program interface to underlying system applications.

Claims

1. A method comprising:

generating, via a processor, from an input lattice representing a first modality of an input, a first markup-language expression;

generating, via a finite-state transducer, a concatenated mapping, wherein the finite-state transducer;

uses the input lattice and a first finite-state machine having associated markup-language semantics to relate the first markup-language expression to a second markup-language expression representing a second modality based on a level of coincidence between the first modality and the second modality, to yield a mapping; and

concatenates input symbols of the mapping associated with the markup-language semantics, to yield the concatenated mapping;

generating, using the input and a second finite-state machine representing the concatenated mapping, a third markup-language expression; and

outputting the third markup-language expression.

2. The method of claim 1, further comprising a finite-state machine representing a regular grammar comprising markup language tags in a terminal of the regular grammar.

3. The method device of claim 1, further comprising a finite-state device representing a regular grammar comprising markup-language tags in a result-part of a terminal of the regular grammar.

4. The method of claim 2, further comprising:

generating a result-finite-state machine representing a multimodal integration by relating a set of input symbols to a set of output symbols, wherein the input lattice is associated with the multimodal integration.

5. The method of claim 2, further comprising converting a sequence of words into a sequence of markup-language symbols using the third markup-language expression.

6. The method of claim 1, wherein the input comprises a command received via at least one of a speech input subsystem, a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem, and a body motion input subsystem.

7. The method of claim 1, further comprising using extensible-markup-language-based symbols in the first finite state machine.

8. The method of claim 1, further comprising using the third markup-language expression to perform another function.

9. A system comprising:

a processor; and

a computer-readable medium storing instructions for controlling the processor to perform a method comprising;

generating, from an input lattice representing a first modality of an input, a first markup-language expression;

generating, via a finite-state transducer, a concatenated mapping, wherein the finite-state transducer;

uses the input lattice and a first finite-state machine having associated markup-language semantics to relate the first markup-language expression to a second markup-language expression representing a second modality based on a level of coincidence between the first modality and the second modality, to yield a mapping; and

concatenates input symbols of the mapping associated with the markup-language semantics, to yield the concatenated mapping;

generating, using the input and a second finite-state machine representing the concatenated mapping, a third markup-language expression; and

outputting the third markup-language expression.

10. The system of claim 9, wherein the finite-state transducer further uses a regular grammar comprising markup-language tags.

11. The system of claim 10, wherein the finite-state transducer further uses a regular grammar comprising markup-language tags in a result-part of a terminal of the regular grammar.

12. The system according to claim 10, wherein the computer-readable storage medium stores further instructions which result in the method further comprising:

generating a result-finite-state machine relating a set of input symbols to a set of output symbols representing a multimodal integration, wherein the input lattice is used for the multimodal integration.

13. The system of claim 9, wherein the input comprises a command received via at least one of a speech input subsystem, a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem, and a body motion input subsystem.

14. The system of claim 9, wherein the finite-state transducer further uses extensible-markup-language-based symbols.

15. The system of claim 9, wherein the computer-readable storage medium stores further instructions which result in the method further comprising using the third markup-language expression to perform another function.

16. A computer-readable storage medium storing instructions which, when executed by a computing device, cause the computing device to perform a method comprising:

generating, from an input lattice representing a first modality of an input, a first markup-language expression;

generating, via a finite-state transducer, a concatenated mapping, wherein the finite state transducer;

uses the input lattice and a first finite-state machine having associated markup-language semantics to relate the first markup-language expression to a second markup-language expression representing a second modality based on a level of coincidence between the first modality and the second modality, to yield a mapping; and

concatenates input symbols of the mapping associated with the markup-language semantics, to yield the concatenated mapping;

generating, using a second finite-state machine representing the concatenated mapping and the input, a third markup-language expression; and

outputting the third markup-language expression.

17. The computer-readable storage medium of claim 16, wherein generating the concatenated mapping further comprises using a finite-state device representing a regular grammar comprising markup language tags in a terminal of the regular grammar.

18. The computer-readable storage medium of claim 17, wherein generating the mapping further comprises using the finite-state device in a result-part of the terminal of the regular grammar.

19. The computer-readable storage medium of claim 17, the computer-readable storage medium storing additional instructions which result in the method further comprising generating a result-finite-state machine relating a set of input symbols to a set of output symbols representing a multimodal integration, wherein the multimodal integration uses the input lattice.

20. The computer-readable storage medium of claim 16, wherein the input comprises a command received via at least one of a speech input subsystem, a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem, and a body motion input subsystem.

21. The computer-readable storage medium of claim 16, wherein using the first finite-state device and the second finite-state device to perform a function using the input lattice comprises using extensible-markup-language-based symbols in the finite state device having symbols based on markup-language semantics.

22. The computer-readable storage medium of claim 16, the computer-readable storage medium storing additional instructions which result in the method further comprising using the generated markup-language expression to another function.