Method and system for resolving cross-modal references in user inputs
First Claim
1. A method for resolving cross-modal references in user inputs to a data processing system, the user inputs being entered through at least one input modality, the method comprising:
- generating a set of multimodal interpretations (MMIs) based on the user inputs collected during a turn, at least one MMI comprising at least one reference, each reference comprising at least one reference variable;
generating one or more sets of joint MMIs, each set of joint MMIs comprising MMIs of semantically compatible types;
generating one or more sets of reference resolved MMIs by resolving reference variables of references of the one or more sets of joint MMIs; and
generating an integrated MMI for each set of reference resolved MMIs, wherein the generation of the integrated MMI is done by unifying the set of reference resolved MMIs.
1 Assignment
0 Petitions
Accused Products
Abstract
A method and a system for resolving cross-modal references in user inputs to a data processing system (100) are provided. The method includes generating (502) a set of multimodal interpretations (MMIs), based on the user inputs collected during a turn. The set of MMIs includes at least one reference, and each reference includes at least one reference variable. The method further includes generating (504) one or more sets of joint MMIs. Each set of joint MMIs includes MMIs of semantically compatible types. The method further includes generating (506) one or more sets of reference-resolved MMIs, by resolving the reference variables of the references contained in the sets of joint MMIs. The method further includes generating (508) an integrated MMI for each set of reference resolved MMIs. The generation of an integrated MMI is carried out by unifying the MMIs in a set of reference resolved MMIs.
182 Citations
21 Claims
-
1. A method for resolving cross-modal references in user inputs to a data processing system, the user inputs being entered through at least one input modality, the method comprising:
-
generating a set of multimodal interpretations (MMIs) based on the user inputs collected during a turn, at least one MMI comprising at least one reference, each reference comprising at least one reference variable;
generating one or more sets of joint MMIs, each set of joint MMIs comprising MMIs of semantically compatible types;
generating one or more sets of reference resolved MMIs by resolving reference variables of references of the one or more sets of joint MMIs; and
generating an integrated MMI for each set of reference resolved MMIs, wherein the generation of the integrated MMI is done by unifying the set of reference resolved MMIs. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)
-
-
16. A method for resolving cross-modal references in user inputs to a data processing system, the user inputs being entered through at least one input modality, the data processing system generating references based on each user input, each reference comprising at least one reference variable, the method comprising:
-
collecting multimodal interpretations (MMIs) corresponding to the user inputs for a user turn;
classifying the collected MMIs into one or more sets of semantically compatible MMIs;
identifying MMIs that comprise one or more references in each of the one or more sets of semantically compatible MMIs;
creating one or more reference association structures (RASs) for each set of semantically compatible MMIs, one RAS for each unique type of MMI required to resolve the references in the identified MMIs with the set of semantically compatible MMIs;
mapping the reference variables of the references in the identified MMIs of a set of semantically compatible MMIs to the one or more RASs contained in that set of semantically compatible MMIs, the mapping being based on the type of MMI required by the reference variable;
sorting the reference variables within each RAS using one or more pre-determined criteria;
mapping each referent, which is an MMI that does not have reference variables, of a set of semantically compatible MMIs to an RAS contained in the set of semantically compatible MMIs requiring referents that are of the same type or super type as the referent;
sorting the referents in each RAS using the one or more pre-determined criteria; and
binding the reference variables in each RAS to one or more referents in the RAS.
-
-
17. A method for resolving cross-modal references in user inputs to a data processing system, the user inputs being entered through at least one input modality, the data processing system generating references based on each user input, each reference comprising at least one reference variable, the method comprising:
-
segmenting the user inputs, wherein the segmenting comprises collecting a set of multimodal interpretations (MMIs) corresponding to the user inputs for a user turn;
classifying the collected set of MMIs semantically, wherein semantically classifying the collected set of MMIs comprises creating sets of joint MMIs, each set of joint MMIs comprising MMIs of semantically compatible types;
resolving the reference variables in the sets of joint MMIs to create corresponding sets of reference-resolved MMIs, wherein resolving the reference variables comprises replacing each reference variable with a resolved value; and
integrating the set of reference-resolved MMIs to generate a corresponding set of integrated MMIs. - View Dependent Claims (18, 19)
-
-
20. An electronic equipment that resolves cross-modal references in user inputs to a data processing system, the user inputs being entered through at least one input modality, the equipment comprising:
-
means for generating a set of multimodal interpretations (MMIs) based on the user inputs collected during a turn, at least one MMI comprising at least one reference, each reference comprising at least one reference variable;
means for generating one or more sets of joint MMIs, each set of joint MMIs comprising MMIs of semantically compatible types;
means for generating a set of reference resolved MMIs for each set of joint MMIs, wherein the generation of the set of reference resolved MMIs is done by resolving reference variables of the references of the set of joint MMIs; and
means for generating an integrated MMI for each set of reference resolved MMIs, wherein the generation of the integrated MMI is done by unifying the set of reference resolved MMIs.
-
-
21. A computer program product for use with a computer, the computer program product comprising a computer usable medium having a computer readable program code embodied therein for resolving cross-modal references in user inputs to a data processing system, the user inputs being entered through at least one input modality, the computer program code performing:
-
generating a set of multimodal interpretations (MMIs) based on the user inputs collected during a turn, at least one MMI comprising at least one reference, each reference comprising at least one reference variable;
generating one or more sets of joint MMIs, each set of joint MMIs comprising MMIs of semantically compatible types;
generating a set of reference resolved MMIs for each set of joint MMIs, wherein the generation of a set of reference resolved MMIs is done by resolving the reference variables of the references of the set of joint MMIs; and
generating an integrated MMI for each set of reference resolved MMIs, wherein the generation of the integrated MMI is done by unifying the set of reference resolved MMIs.
-
Specification