Model Based Approach for On-Screen Item Selection and Disambiguation

• US 20150248886A1
• Filed: 03/03/2014
• Published: 09/03/2015
• Est. Priority Date: 03/03/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sarikaya, Ruhi, Celikyilmaz, Fethiye Asli, Heck, Larry Paul, Hakkani-Tur, Dilek Z., Feizollahi, Zhaleh
• Granted Patent: US 9,412,363 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A model-based approach for on-screen item selection and disambiguation is provided. An utterance may be received by a computing device in response to a display of a list of items for selection on a display screen. A disambiguation model may then be applied to the utterance. The disambiguation model may be utilized to determine whether the utterance is directed to at least one of the list of displayed items, extract referential features from the utterance and identify an item from the list corresponding to the utterance, based on the extracted referential features. The computing device may then perform an action which includes selecting the identified item associated with utterance.

Claims

1. A method comprising:

receiving, by a computing device, an utterance in response to displaying a plurality of items on a display;

applying, by the computing device, a disambiguation model to the utterance for;

identifying whether the utterance is directed to at least one of the plurality of items on the display;

extracting a plurality of referential features from the utterance; and

identifying an item among the plurality of items on the display corresponding to the utterance based on the plurality of referential features extracted from the utterance; and

performing, by the computing device, an action associated with the utterance upon identifying the item among the plurality of items on the display corresponding to the utterance.

2. The method of claim 1, wherein identifying whether the utterance is directed at least one of the plurality of items on the display comprises:

building a statistical classifier model comprising lexical and semantic features, the lexical and semantic features comprising one or more of the following;

a vocabulary obtained from text in the utterance, a phrase match between the utterance and item metadata associated with the plurality of items on the display, and locational features; and

applying the statistical classifier model to the utterance.

3. The method of claim 1, wherein identifying an item among the plurality of items on the display based on the plurality of referential features extracted from the utterance comprises:

identifying an explicit reference to the item based on a portion of text in the utterance;

identifying an explicit reference to the item based on a textual cue in the utterance; and

identifying an explicit reference to the item based on screen location data in the utterance.

4. The method of claim 3, wherein identifying an item among the plurality of items on the display based on the plurality of referential features extracted from the utterance further comprises:

identifying an implicit reference to the item based on the utterance; and

identifying an implicit reference to a position of the item based on the utterance.

5. The method of claim 4, wherein identifying an item among the plurality of items on the display corresponding to the utterance further comprises:

determining a plurality of match scores between the utterance and metadata associated with each of the plurality of items on the display;

parsing the utterance for matching phrases between semantic phrases in the utterance and the metadata; and

parsing the utterance to capture location indicators for predicting a screen location of the item.

6. The method of claim 1, wherein performing, by the computing device, an action associated with the utterance upon identifying the item among the plurality of items on the display corresponding to the utterance, comprises:

selecting the item on the display; and

displaying additional information about the selected item.

7. The method of claim 1, wherein performing, by the computing device, an action associated with the utterance upon identifying the item among the plurality of items on the display corresponding to the utterance, comprises:

selecting the item on the display; and

executing an activity associated with the selected item.

8. A computing device comprising:

a memory for storing executable program code; and

a processor, functionally coupled to the memory, the processor being responsive to computer-executable instructions contained in the program code and operative to;

receive an utterance in response to displaying a plurality of items on a display screen;

apply a disambiguation model to the utterance, the disambiguation being utilized to;

identify whether the utterance is directed to at least one of the plurality of items on the display;

extract a plurality of referential features from the utterance; and

identify an item among the plurality of items on the display corresponding to the utterance based on the plurality of referential features extracted from the utterance; and

perform an action associated with the utterance upon identifying the item among the plurality of items on the display corresponding to the utterance.

9. The computing device of claim 8, wherein the processor, in identifying whether the utterance is directed at least one of the plurality of items on the display, is operative to:

build a statistical classifier model comprising lexical and semantic features, the lexical and semantic features comprising one or more of the following;

a vocabulary obtained from text in the utterance, a phrase match between the utterance and item metadata associated with the plurality of items on the display, and locational features; and

apply the statistical classifier model to the utterance.

10. The computing device of claim 8, wherein the processor, in identifying an item among the plurality of items on the display based on the plurality of referential features extracted from the utterance, is operative to:

identify an explicit reference to the item based on a portion of text in the utterance;

identify an explicit reference to the item based on a textual cue in the utterance; and

identify an explicit reference to the item based on screen location data in the utterance.

11. The computing device of claim 10, wherein the processor, in identifying an item among the plurality of items on the display based on the plurality of referential features extracted from the utterance, is further operative to:

identify an implicit reference to the item based on the utterance; and

identify an implicit reference to a position of the item based on the utterance.

12. The computing device of claim 11, wherein the processor, in identifying an item among the plurality of items on the display corresponding to the utterance, is further operative to:

determine a plurality of match scores between the utterance and metadata associated with each of the plurality of items on the display;

parse the utterance for matching phrases between semantic phrases in the utterance and the metadata; and

parse the utterance to capture location indicators for predicting a screen location of the item.

13. The computing device of claim 8, wherein the processor, in performing an action associated with the utterance upon identifying the item among the plurality of items on the display corresponding to the utterance, is operative to:

select the item on the display; and

display additional information about the selected item.

14. The computing device of claim 8, wherein the processor, in performing an action associated with the utterance upon identifying the item among the plurality of items on the display corresponding to the utterance, is operative to:

select the item on the display; and

execute an activity associated with the selected item.

15. A computer-readable storage medium storing computer executable instructions which, when executed by a computer, will cause computer to perform a method comprising:

receiving a first utterance comprising a query for a plurality of items;

displaying the plurality of items in response to the first utterance on a display;

receiving a second utterance in response to displaying the plurality of items on the display;

applying a disambiguation model to the second utterance for;

identifying whether the second utterance is directed to at least one item in the plurality of items on the display;

extracting a plurality of referential features from the second utterance; and

identifying an item among the plurality of items on the display corresponding to the second utterance based on the plurality of referential features extracted from the second utterance; and

performing, by the computing device, an action associated with the second utterance upon identifying the item among the plurality of items on the display corresponding to the second utterance.

16. The computer-readable storage medium of claim 15, wherein identifying whether the second utterance is directed at least one of the plurality of items on the display comprises:

building a statistical classifier model comprising lexical and semantic features, the lexical and semantic features comprising one or more of the following;

a vocabulary obtained from text in the second utterance, a phrase match between the second utterance and item metadata associated with the plurality of items on the display, and locational features; and

applying the statistical classifier model to the second utterance.

17. The computer-readable storage medium of claim 15, wherein identifying an item among the plurality of items on the display based on the plurality of referential features extracted from the second utterance comprises:

identifying an explicit reference to the item based on a portion of text in the second utterance;

identifying an explicit reference to the item based on a textual cue in the second utterance; and

identifying an explicit reference to the item based on screen location data in the second utterance.

18. The computer-readable storage medium of claim 17, wherein identifying an item among the plurality of items on the display based on the plurality of referential features extracted from the second utterance further comprises:

identifying an implicit reference to the item based on the second utterance; and

identifying an implicit reference to a position of the item based on the second utterance.

19. The computer-readable storage medium of claim 18, wherein identifying an item among the plurality of items on the display corresponding to the second utterance further comprises:

determining a plurality of match scores between the second utterance and metadata associated with each of the plurality of items on the display;

parsing the second utterance for matching phrases between semantic phrases in the second utterance and the metadata; and

parsing the second utterance to capture location indicators for predicting a screen location of the item.

20. The computer-readable storage medium of claim 15, wherein performing, by the computing device, an action associated with the second utterance upon identifying the item among the plurality of items on the display corresponding to the second utterance, comprises:

selecting the item on the display;

displaying additional information about the selected item; and

executing an activity associated with the selected item.