Systems and methods for extracting meaning from multimodal inputs using finite-state devices

• US 8,355,916 B2
• Filed: 05/31/2012
• Issued: 01/15/2013
• Est. Priority Date: 07/12/2001
• Status: Expired due to Term

• Current Assignee: Interactions, LLC
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Bangalore, Srinivas, Johnston, Michael J.
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US13/485,574
• Publication Number:
  US 20120303370A1
• Time in Patent Office: 229 Days
• Field of Search: 704/9, 704/251, 704/256, 704/243, 704/255, 704/270, 706/11, 715/809, 715/863
• US Class Current: 704/249
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features

Abstract

Multimodal utterances contain a number of different modes. These modes can include speech, gestures, and pen, haptic, and gaze inputs, and the like. This invention use recognition results from one or more of these modes to provide compensation to the recognition process of one or more other ones of these modes. In various exemplary embodiments, a multimodal recognition system inputs one or more recognition lattices from one or more of these modes, and generates one or more models to be used by one or more mode recognizers to recognize the one or more other modes. In one exemplary embodiment, a gesture recognizer inputs a gesture input and outputs a gesture recognition lattice to a multimodal parser. The multimodal parser generates a language model and outputs it to an automatic speech recognition system, which uses the received language model to recognize the speech input that corresponds to the recognized gesture input.

Claims

1. An apparatus for recognizing an utterance comprising:

means for receiving an utterance, the utterance comprising a first portion having a first mode and a second portion having a second mode;

means for generating a first mode recognition lattice and a first finite-state transducer, based on the first mode;

means for relating the first portion of the utterance to the second portion of the utterance based on the first finite-state transducer; and

means for generating a second finite-state transducer, the second finite-state transducer comprising a gesture and speech recognition model finite-state transducer, based on the first mode recognition lattice and the first finite-state transducer.

2. The apparatus of claim 1, further comprising:

means for recognizing the second mode based on the second finite-state transducer.

3. The apparatus of claim 1 wherein the first mode is a gesture mode.

4. The apparatus of claim 1 wherein the second mode is a speech mode.

5. The apparatus of claim 1 wherein the generating the first mode recognition lattice is further based on a first mode feature lattice.

6. An apparatus for recognizing an utterance comprising:

means for receiving an utterance comprising a plurality of modes;

means for relating a first portion of the utterance comprising a first mode to a second portion of the utterance comprising a second mode, based on a first finite-state transducer; and

means for generating a second finite-state transducer, the second finite-state transducer comprising a gesture and speech recognition model finite-state transducer, based on a first mode recognition lattice and the first finite-state transducer.

7. The apparatus of claim 6 further comprising:

means for generating the first mode recognition lattice and the first finite-state transducer, based on the first mode.

8. The apparatus of claim 7 further comprising:

means for recognizing the second mode based on the second finite-state transducer.

9. The apparatus of claim 7 wherein the first mode is a gesture mode.

10. The apparatus of claim 7 wherein the second mode is a speech mode.

11. The apparatus of claim 7 wherein the generating the first mode recognition lattice is further based on a first mode feature lattice.

12. The apparatus of claim 11 further comprising:

means for generating the first mode feature lattice based on the utterance.

13. A apparatus for extracting meaning from multimodal inputs comprising:

means for receiving a multimodal input;

means for relating a first portion of the multimodal input comprising a first mode to a second portion of the multimodal input comprising a second mode, based on a first finite-state transducer; and

means for generating a second finite-state transducer, the second finite-state transducer comprising a gesture and speech recognition model finite-state transducer, based on a first mode recognition lattice and the first finite-state transducer.

14. The apparatus of claim 13 further comprising:

means for generating the first mode recognition lattice and the first finite-state transducer, based on the first mode.

15. The apparatus of claim 14 wherein the first mode is a gesture mode.

16. The apparatus of claim 14 wherein the second mode is a speech mode.

17. The apparatus of claim 14 wherein the generating the first mode recognition lattice is further based on a first mode feature lattice.

18. The apparatus of claim 17 further comprising:

means for generating the first mode feature lattice based on the multimodal input.