Systems and methods for generating markup-language based expressions from multi-modal and unimodal inputs

• US 7,257,575 B1
• Filed: 10/24/2002
• Issued: 08/14/2007
• Est. Priority Date: 10/24/2002
• Status: Expired due to Fees

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Johnston, Michael J., Bangalore, Srinivas
• Primary Examiner(s): HWANG, JOON H
• Application Number:
  US10/278,921
• Time in Patent Office: 1,755 Days
• Field of Search: 707 1- 10, 707100-1041, 704 1- 10
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/151 Transformation
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

When using finite-state devices to perform various functions, it is beneficial to use finite state devices representing regular grammars with terminals having markup-language-based semantics. By using markup-language-based symbols in the finite state devices, it is possible to generate valid markup-language expressions by concatenating the symbols representing the result of the performed function. The markup-language expression can be used by other applications and/or devices. Finite-state devices are used to convert strings of words and gestures into valid markup-language, for example, XML, expressions that can be used, for example, to provide an application program interface to underlying system applications.

Claims

1. A method for generating markup-language-based expressions from an input into an unimodal or a multimodal device, comprising:

receiving at least one data input in at least one input mode of a recognition device;

generating at least one input lattice representing the at least one data input;

using at least one finite-state device to perform a function using the at least one generated input lattice, the finite-state device having symbols based on markup-language semantics, wherein performing the function comprises mapping a first markup-language expression to a second markup-language expression using a transducer relating the first-markup-language expression to the second-markup-language expression;

generating a result finite-state device representing a result of the performed function, the result finite-state device relating at least one set of input symbols to at least one set of output symbols;

concatenating output symbols in the at least one set of output symbols of the result finite-state device to generate at least one markup-language expression;

generating the at least one markup-language expression; and

outputting the at least one generated markup-language expression to a processor.

2. The method of claim 1, wherein using at least one finite-state device to perform a function using the at least one generated input lattice comprises using at least one finite-state device representing a regular grammar containing markup language tags in at least one terminal of the regular grammar.

3. The method of claim 2, wherein using at least one finite-state device representing a regular grammar containing markup-language tags in at least one terminal of the regular grammar comprises using at least one finite-state device representing a regular grammar containing markup-language tags in a result-part of at least one terminal of the regular grammar.

4. The method according to claim 2, wherein:

using at least one finite-state device to perform a function using the at least one generated input lattice comprises performing multimodal integration using the at least one generated input lattice; and

generating a result-finite-state device representing a result of the performed function comprises generating a result-finite-state device relating at least one set of input symbols to at least one set of output symbols representing the multimodal integration.

5. The method according to claim 2, wherein using at least one finite-state device to perform a function using the at least one generated input lattice comprises converting a sequence of words into a sequence of markup-language symbols using a finite-state transducer relating words to markup-language symbols.

6. The method of claim 1, wherein receiving at least one data input in at least one input mode of a recognition device comprises inputting at least one of a command and information using at least one of a speech input subsystem, a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem, and a body motion input subsystem.

7. The method according to claim 1, wherein using at least one finite-state device to perform a function using the at least one generated input lattice comprises using extensible-markup-language-based symbols in the finite state device having symbols based on markup-language semantics.

8. The method according to claim 1, further comprising using the at least one generated markup-language expression to perform at least one other function.

9. A method for generating markup-language-based expressions from an input into an unimodal device or a multimodal device, comprising:

receiving at least one data input in at least one input mode of a recognition device;

generating at least one input lattice representing the at least one data input;

using at least one finite-state transducer relating a first set of symbols to a second set of symbols to perform a function using the at least one generated input lattice, the second set of symbols having markup-language-based symbols, wherein performing the function further comprises mapping a first-markup-language expression to a second-markup-language expression using a transducer relating the first-markup-language expression to the second-markup-language expression;

generating a result finite-state device representing a result of the performed function, the result finite-state device relating at least one set of input symbols to at least one set of output symbols;

concatenating output symbols in the at least one set of output symbols of the result finite-state device to generate at least one markup-language expression;

generating the at least one markup-language expression; and

outputting the at least one generated markup-language expression to a processor.

10. The method of claim 9, wherein using at least one finite-state transducer relating a first set of symbols to a second set of symbols comprises using at least one finite-state transducer representing a regular grammar containing markup-language tags.

11. The method of claim 10, wherein using at least one finite-state transducer representing a regular grammar containing markup-language tags comprises using at least one finite-state transducer representing a regular grammar containing markup-language tags in a result-part of at least one terminal of the regular grammar.

12. The method according to claim 10, wherein:

using at least one finite-state transducer relating a first set of symbols to a second set of symbols comprises performing multimodal integration using the at least one generated input lattice; and

generating a result-finite-state device representing a result of the performed function comprises generating a result-finite-state device relating at least one set of input symbols to at least one set of output symbols representing the multimodal integration.

13. The method of claim 9, wherein receiving at least one data input in at least one input mode of a recognition device comprises inputting at least one of a command and information using at least one of a speech input subsystem, a pen input subsystem, a computer vision input subsystem, a haptic input subsystem, a gaze input subsystem, and a body motion input subsystem.

14. The method according to claim 9, wherein using at least one finite-state transducer relating a first set of symbols to a second set of symbols to perform a function using the at least one generated input lattice, the at least one finite-state transducer having symbols based on markup-language semantics, further comprises using extensible-markup-language-based symbols in the at least one finite state transducer having symbols based on markup-language semantics.

15. The method according to claim 9, further comprising using the at least one generated markup-language expression to perform at least one other function.