Language and domain independent model based approach for on-screen item selection

• US 9,886,958 B2
• Filed: 12/11/2015
• Issued: 02/06/2018
• Est. Priority Date: 12/11/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Celikyilmaz, Fethiye Asli, Feizollahi, Zhaleh, Hakkani-Tur, Dilek, Sarikaya, Ruhi
• Primary Examiner(s): Godbold, Douglas
• Application Number:
  US14/966,257
• Publication Number:
  US 20170169829A1
• Time in Patent Office: 788 Days
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G10L 17/04 Training, enrolment or model building
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 17/26 Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices

Abstract

A universal model-based approach for item disambiguation and selection is provided. An utterance may be received by a computing device in response to a list of items for selection. In aspects, the list of items may be displayed on a display screen. The universal disambiguation model may then be applied to the utterance. The universal disambiguation model may be utilized to determine whether the utterance is directed to at least one of the list of items and identify an item from the list corresponding to the utterance, based on identified language and/or domain independent referential features. The computing device may then perform an action which may include selecting the identified item associated with utterance.

Claims

1. A computing device comprising:

at least one processor; and

a memory for storing and encoding computer executable instructions that, when executed by the at least one processor are operative to;

receive an utterance from a user;

extract features from the utterance to form extracted features;

identify that the utterance is directed to at least one of a plurality of items previously provided by the computing device to the user utilizing a language independent disambiguation model, wherein the language independent disambiguation model identifies that the utterance is directed to the at least one of the plurality of items by;

identifying as universal features one or more domain independent features and language independent features in the extracted features;

determining an overlap between one or more universal features extracted from the utterance and one or more features associated with the plurality of items, wherein the one or more features are identified based on metadata associated with the plurality of items; and

identifying the at least one of the plurality of items corresponding to the utterance based on the overlap; and

send instructions to perform an action associated with the utterance upon identifying that the utterance is directed to the at least one of the plurality of items.

2. The computing device of claim 1, wherein the utterance is in a first language, and wherein the language independent disambiguation model is trained utilizing data from a different language from the first language.

3. The computing device of claim 1, wherein the plurality of items is provided by being presented on a display.

4. The computing device of claim 1, wherein features associated with each of the plurality of items are identified from metadata associated with each of the plurality of items.

5. The computing device of claim 1, wherein extracting the features from the utterance further comprises:

determining one or more relations between features of the utterance and features associated with one or more of the plurality of items; and

extracting the features from the utterance having a relation to the one or more features associated with one or more items of the plurality of items.

6. The computing device of claim 5, wherein a plurality of language understanding models are utilized to extract the features from the utterance having the relation to the one or more features associated with the one or more items of the plurality of items, and wherein each of the plurality of language understanding models is trained utilizing data from a different language.

7. The computing device of claim 6, wherein the utterance is in a first language that is different from any languages utilized to train the plurality of language understanding models.

8. The computing device of claim 1, wherein determining the overlap further comprises:

determining a lexical overlap between at least one word of the utterance and at least one word in a title or property of the at least one of the plurality of items.

9. The computing device of claim 1, wherein the plurality of items is presented on a display, and wherein determining the overlap further comprises:

building a statistical classifier model comprising lexical and semantic features, the lexical and semantic features comprising one or more of the following;

a match between the utterance and metadata associated with one or more of the plurality of items on the display and a match between the utterance and locational features associated with one or more of the plurality of items on the display; and

applying the statistical classifier model to the utterance.

10. The computing device of claim 8, wherein determining the lexical overlap further comprises one or more of:

identifying an explicit reference to the at least one item based on a portion of text in the utterance;

identifying an explicit reference to the at least one item based on a textual cue in the utterance; and

identifying an explicit reference to the at least one item based on screen location data in the utterance.

11. The computing device of claim 8, wherein determining the lexical overlap further comprises:

identifying an implicit reference to the at least one item based on the utterance; and

identifying an implicit reference to a position of the at least one item based on the utterance.

12. The computing device of claim 1, wherein the plurality of items is presented on a display, and wherein determining the overlap further comprises:

determining a plurality of match scores between the universal features extracted from the utterance and the features associated with one or more of the plurality of items on the display;

parsing the utterance for matching phrases between semantic words or phrases in the utterance and words or phrases in a title or property of the at least one of the plurality of items; and

parsing the utterance to capture location indicators for predicting a screen location of the at least one item of the plurality of items.

13. The computing device of claim 1, wherein the action comprises:

selecting the at least one item; and

providing additional information about the selected at least one item.

14. The computing device of claim 1, wherein the action comprises:

selecting the at least one item; and

executing an activity associated with the selected at least one item.

15. The computing device of claim 1, wherein the language independent disambiguation model is trained for a specific application, wherein the specific application is at least one of:

a digital assistant application;

a voice recognition application;

an email application;

a social networking application;

a collaboration application;

an enterprise management application;

a messaging application;

a word processing application;

a spreadsheet application;

a database application;

a presentation application;

a contacts application;

a gaming application;

an e-commerce application;

an e-business application;

a transactional application;

an exchange application; and

a calendaring application.

16. A method comprising:

displaying a plurality of items on a display;

receiving, by a computing device, an utterance in a first language in response to the displaying the plurality of items;

extracting, by the computing device, features from the utterance having a relation to features associated with one or more of the plurality of items to form extracted features;

identifying, by the computing device, that the utterance is directed to at least one of the plurality of items on the display utilizing a language independent disambiguation model,

wherein the language independent disambiguation model is trained with data from a second language and does not require any training data from the first language, and

wherein the language independent disambiguation model identifies that the utterance is directed to the at least one of the plurality of items on the display by;

identifying domain and language independent features in the extracted features;

selecting the domain and language independent features from the extracted features to form selected features;

determining an overlap between the selected features extracted from the utterance and the features associated with one or more of the plurality of items on the display; and

identifying that the at least one of the plurality of items on the display corresponds to the utterance based on the overlap; and

performing, by the computing device, an action associated with the utterance in response to identifying that the utterance is directed to the at least one of the plurality of items on the display.

17. The method of claim 16, wherein the first language is different than the second language.

18. The method of claim 16, wherein determining the overlap comprises:

determining a lexical overlap between at least one of the selected features extracted from the utterance and at least one of the features associated with the at least one of the plurality of items on the display.

19. The method of claim 16, wherein the computing device is at least one of:

a mobile telephone;

a smart phone;

a tablet;

a smart watch;

a wearable computer;

a personal computer;

a desktop computer;

a gaming system; and

a laptop computer.

20. A computer readable storage device storing computer executable instructions which, when executed by a computing device, cause the computing device to perform a method comprising:

sending instructions to present a plurality of items;

receiving, by the computing device, an utterance in a first language;

extracting, by the computing device, features from the utterance utilizing a language understanding model to form extracted features;

determining, by the computing device, that the utterance is directed to at least one of the plurality of items by utilizing a language independent flexible item selection model,

wherein the language independent flexible item selection model has not received training data for the first language,

wherein the language independent flexible item selection model determines that the utterance is directed to the at least one of the plurality of items by;

identifying as universal features one or more domain independent features and language independent features in the extracted features; and

determining a lexical overlap between one or more universal features extracted from the utterance and one or more features associated with the at least one of the plurality of items; and

sending instructions to perform an action associated with the utterance upon determining that the utterance is directed to the at least one item among the plurality of items.