Systems and methods for aggregating related inputs using finite-state devices and extracting meaning from multimodal inputs using aggregation

• US 20030055644A1
• Filed: 08/12/2002
• Published: 03/20/2003
• Est. Priority Date: 08/17/2001
• Status: Abandoned Application

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Bangalore, Srinivas, Johnston, Michael J.
• Application Number:
  US10/216,448
• Publication Number:
  US 20030055644A1
• Time in Patent Office: Days
• US Class Current: 704/251
• CPC Class Codes: G01C 21/3664 Details of the user input interface, e.g. buttons, knobs or sliders, including those provided on a touch screen; remote controllers; input using gestures
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 8/35 model driven
• CPC Class Codes: G06F 8/37 Compiler construction; Parser generation
• CPC Class Codes: G06V 30/1985 Syntactic analysis, e.g. using a grammatical approach syntactic image representation G06V30/18171
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/32 Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems

Abstract

When using finite-state methods to represent inputs, it is beneficial to combine related inputs and to represent combinations of related inputs in the finite-state device. In multimodal recognition systems and methods that integrate different recognized modes, combining related inputs helps the unification process.

Claims

1. A method for combining related inputs of a single mode of a multimodal recognition device, comprising;

receiving a plurality of inputs of the single mode of an input portion of the multimodal device;

generating an input lattice from the plurality of inputs;

combining at least two related inputs in the input lattice; and

adding at least one branch to the input lattice, wherein the added branch represents the combination of the related inputs.

2. The method of claim 1, wherein:

combining related inputs in the input lattice comprises combining related inputs in the input lattice; and

the related inputs are temporally and/or spatially related.

3. The method of claim 1, wherein, for each of the at least two related inputs, receiving those inputs comprises receiving a selection, by a user, of at least one of an entity on the input portion, an area on the input portion and a location on the input portion.

4. The method of claim 3, wherein receiving the selection of the entity includes receiving a selection, by the user, of at least one of at least one object and at least one command on the input portion.

5. The method of claim 2, wherein combining related inputs comprises combining related inputs that represent at least two types of entities.

6. The method of claim 2, wherein:

combining related inputs comprises combining related inputs that represent entities of a same type.

7. The method of claim 2, further comprising maintaining a record of the received plurality of inputs.

8. The method of claim 7, wherein maintaining the record of the received plurality of inputs comprises assigning at least one different weight to the received plurality of inputs than a weight assigned to the combined related inputs.

9. A method for combining adjacent inputs of a multimodal device comprising;

receiving a plurality of inputs of one mode of the multimodal device in an input portion of the multimodal device;

generating an input lattice from the plurality of inputs;

combining at least two adjacent inputs on the input lattice; and

adding at least one branch to the input lattice, wherein the added branch represents the combination of the adjacent inputs.

10. The method of claim 9, wherein combining adjacent inputs on the input lattice comprises combining adjacent inputs on the input lattice that are temporally related and/or spatially related.

11. The method of claim 10, wherein, for each of the at least two adjacent inputs, receiving those inputs comprises receiving a selection, by a user, of at least one of an entity on the input portion of the multimodal device, an area on the input portion of the multimodal device and a location on the input portion of the multimodal device.

12. The method of claim 11, wherein receiving the selection of the entity includes receiving a selection, by the user, of at least one of at least one object and at least one command on the input portion of the multimodal device.

13. The method of claim 11, wherein combining adjacent inputs comprises combining adjacent inputs that represent at least two types of entities.

14. The method of claim 11, wherein combining adjacent inputs comprises combining adjacent inputs that represent entities of a same type.

15. The method of claim 10, further comprising maintaining a record of the received plurality of inputs.

16. The method of claim 15, wherein maintaining the record of the received plurality of inputs comprises assigning at least one different weight to the received plurality of inputs than a weight assigned to the combined related inputs.

17. A method for combining related inputs, comprising:

receiving a plurality of inputs in a first mode of a multimodal utterance;

receiving at least one input in a second mode of a multimodal utterance, wherein the at least one input in the second mode of the multimodal utterance contains a reference to a plurality of items;

generating an input lattice from the plurality of inputs in the first mode of the multimodal utterance;

combining at least two related inputs in the first mode of the multimodal utterance; and

adding at least one branch, representing at least one combination of related inputs, to the input lattice.

18. The method of claim 17, wherein combining related inputs in the first mode of the multimodal utterance comprises combining inputs in the input lattice that are temporally-related and/or spatially-related.

19. The method of claim 17, wherein combining the inputs in the first mode of the multimodal utterance comprises combining inputs in the first mode of the multimodal utterance that represent a same type of entity.

20. The method of claim 17, wherein combining the inputs in the first mode of the multimodal utterance comprises combining inputs in the first mode of the multimodal utterance that represent at least two types of entities.

21. The method of claim 17, wherein receiving at least one input in the second mode of the multimodal utterance comprises receiving an utterance in the second mode of the multimodal utterance that includes a reference to at least one of an entity, an area and a location on the input portion that are selected by a user.

22. The method of claim 17, further comprising adding at least one branch to the input lattice representing the plurality of inputs in the first mode of the multimodal utterance.

23. The method of claim 17, further comprising inputting an utterance in the second mode of the multimodal utterance, wherein the utterance includes at least one number representing a number of items to which the utterance in the second mode refers.

24. The method of claim 17, further comprising maintaining a record of the received plurality of inputs.

25. The method of claim 24, wherein maintaining the record of the received plurality of inputs comprises assigning at least one different weight to the received plurality of inputs than a weight assigned to the combined related inputs.