SYSTEM AND METHOD FOR MULTIMODAL INTERACTION USING ROBUST GESTURE PROCESSING

• US 20100281435A1
• Filed: 04/30/2009
• Published: 11/04/2010
• Est. Priority Date: 04/30/2009
• Status: Abandoned Application

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: Johnston, Michael, Bangalore, Srinivas
• Application Number:
  US12/433,320
• Publication Number:
  US 20100281435A1
• Time in Patent Office: Days
• US Class Current: 715/863
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Disclosed herein are systems, computer-implemented methods, and tangible computer-readable media for multimodal interaction. The method includes receiving a plurality of multimodal inputs associated with a query, the plurality of multimodal inputs including at least one gesture input, editing the at least one gesture input with a gesture edit machine. The method further includes responding to the query based on the edited gesture input and remaining multimodal inputs. The gesture inputs can be from a stylus, finger, mouse, and other pointing/gesture device. The gesture input can be unexpected or errorful. The gesture edit machine can perform actions such as deletion, substitution, insertion, and aggregation. The gesture edit machine can be modeled as a finite-state transducer. In one aspect, the method further includes generating a lattice for each input, generating an integrated lattice of combined meaning of the generated lattices, and responding to the query further based on the integrated lattice.

Claims

1. A computer-implemented method of multimodal interaction, the method comprising:

receiving a plurality of multimodal inputs associated with a query, the plurality of multimodal inputs including at least one gesture input;

editing the at least one gesture input with a gesture edit machine; and

responding to the query based on the edited at least one gesture input and the remaining multimodal inputs.

2. The computer-implemented method of claim 1, wherein the at least one gesture input comprises at least one unexpected gesture.

3. The computer-implemented method of claim 1, wherein the at least one gesture input comprises at least one errorful gesture.

4. The computer-implemented method of claim 1, wherein the gesture edit machine performs one or more action selected from a list comprising deletion, substitution, insertion, and aggregation.

5. The computer-implemented method of claim 1, wherein the gesture edit machine is modeled by a finite-state transducer.

6. The computer-implemented method of claim 1, the method further comprising:

generating a lattice for each multimodal input;

generating an integrated lattice which represents a combined meaning of the generated lattices by combining the generated lattices; and

responding to the query further based on the integrated lattice.

7. The computer-implemented method of claim 6, the method further comprising capturing the alignment of the lattices in a single declarative multimodal grammar representation.

8. The computer-implemented method of claim 7, wherein a cascade of finite state operations aligns and integrates content in the lattices.

9. The computer-implemented method of claim 7, the method further comprising compiling the multimodal grammar representation into a finite-state machine operating over each of the plurality of multimodal inputs and over the combined meaning.

10. The computer-implemented method of claim 4, wherein the action of aggregation aggregates one or more inputs of identical type as a single conceptual input.

11. The computer-implemented method of claim 1, wherein the plurality of multimodal inputs are received as part of a single turn of interaction.

12. The computer-implemented method of claim 1, wherein gesture inputs comprise one or more of stylus-based input, finger-based touch input, mouse input, and other pointing device input.

13. The computer-implemented method of claim 1, wherein responding to the request comprises outputting a multimodal presentation that synchronizes one or more of graphical callouts, still images, animation, sound effects, and synthetic speech.

14. The computer-implemented method of claim 1, wherein editing the at least one gesture input with a gesture edit machine is associated with a cost established either manually or via learning based on a multimodal corpus based on the frequency of each edit and further based on gesture complexity.

15. A system for multimodal interaction, the system comprising:

a processor;

a module configured to control the processor to receive a plurality of multimodal inputs associated with a query, the plurality of multimodal inputs including at least one gesture input;

a module configured to control the processor to edit the at least one gesture input with a gesture edit machine; and

a module configured to control the processor to respond to the query based on the edited at least one gesture input and the remaining multimodal inputs.

16. The system of claim 15, wherein the at least one gesture input comprises at least one unexpected gesture.

17. The system of claim 15, wherein the at least one gesture input comprises at least one errorful gesture.

18. The system of claim 15, wherein the gesture edit machine performs one or more action selected from a list comprising deletion, substitution, insertion, and aggregation.

20. The tangible computer-readable medium of claim 18, wherein the gesture edit machine performs one or more action selected from a list comprising deletion, substitution, insertion, and aggregation.

19. A tangible computer-readable medium storing a computer program having instructions for multimodal interaction, the instructions comprising:

receiving a plurality of multimodal inputs associated with a query, the plurality of multimodal inputs including at least one gesture input;

editing the at least one gesture input with a gesture edit machine; and

responding to the query based on the edited at least one gesture input and the remaining multimodal inputs.