Model based approach for on-screen item selection and disambiguation

• US 9,412,363 B2
• Filed: 03/03/2014
• Issued: 08/09/2016
• Est. Priority Date: 03/03/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sarikaya, Ruhi, Celikyilmaz, Fethiye Asli, Feizollahi, Zhaleh, Heck, Larry Paul, Hakkani-Tur, Dilek Z.
• Primary Examiner(s): Baker, Charlotte M
• Application Number:
  US14/194,964
• Publication Number:
  US 20150248886A1
• Time in Patent Office: 890 Days
• Field of Search: 704/235, 704/257, 704/276, 704/251, 704/270, 704/270.1, 725/53
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

A model-based approach for on-screen item selection and disambiguation is provided. An utterance may be received by a computing device in response to a display of a list of items for selection on a display screen. A disambiguation model may then be applied to the utterance. The disambiguation model may be utilized to determine whether the utterance is directed to at least one of the list of displayed items, extract referential features from the utterance and identify an item from the list corresponding to the utterance, based on the extracted referential features. The computing device may then perform an action which includes selecting the identified item associated with utterance.

Claims

1. A method comprising:

storing metadata related to items displayed on a display;

receiving, by a computing device, an utterance;

in response to the received utterance, applying, by the computing device, a disambiguation model to the utterance, including;

identifying whether the utterance is directed to at least one of the plurality of items on the display,

extracting a plurality of referential features from the utterance, and

identifying an item among the plurality of items on the display corresponding to the utterance by comparing at least some of the plurality of referential features extracted from the utterance to the metadata; and

performing, by the computing device, an action associated with the utterance in response to identifying the item corresponding to the utterance.

2. The method of claim 1, wherein identifying whether the utterance is directed to at least one of the plurality of items on the display comprises:

building a statistical classifier model comprising lexical and semantic features, the lexical and semantic features comprising one or more of the following;

a vocabulary obtained from text in the utterance, a phrase match between the utterance and metadata, and locational features; and

applying the statistical classifier model to the utterance.

3. The method of claim 1, wherein identifying an item among the plurality of items on the display comprises:

identifying an explicit reference to the item based on a portion of text in the utterance;

identifying an explicit reference to the item based on a textual cue in the utterance; and

identifying an explicit reference to the item based on screen location data in the utterance.

4. The method of claim 3, wherein identifying an item among the plurality of items on the display comprises:

identifying an implicit reference to the item based on the utterance; and

identifying an implicit reference to a position of the item based on the utterance.

5. The method of claim 4, wherein identifying an item among the plurality of items on the display further comprises:

determining a plurality of match scores between the utterance and the metadata;

parsing the utterance for matching phrases between semantic phrases in the utterance and the metadata; and

parsing the utterance to capture location indicators for predicting a screen location of the item.

6. The method of claim 1, wherein performing, by the computing device, an action associated with the utterance upon identifying the item corresponding to the utterance, comprises:

selecting the item on the display; and

displaying additional information about the selected item.

7. The method of claim 1, wherein performing, by the computing device, an action associated with the utterance upon identifying the item corresponding to the utterance, comprises:

selecting the item on the display; and

executing an activity associated with the selected item.

8. A computing device comprising:

a memory for storing executable program code; and

a processor, functionally coupled to the memory, the processor being responsive to computer-executable instructions contained in the program code and operative to;

store metadata related to items displayed on a display;

receive an utterance;

apply a disambiguation model to the utterance, the disambiguation model being utilized to;

identify whether the utterance is directed to at least one of the plurality of items on the display,

extract a plurality of referential features from the utterance, and

identify an item among the plurality of items on the display corresponding to the utterance by comparing at least some of the plurality of referential features extracted from the utterance to the metadata; and

perform an action associated with the utterance in response to identifying the item corresponding to the utterance.

9. The computing device of claim 8, wherein the processor, in identifying whether the utterance is directed at least one of the plurality of items on the display, is operative to:

build a statistical classifier model comprising lexical and semantic features, the lexical and semantic features comprising one or more of the following;

a vocabulary obtained from text in the utterance, a phrase match between the utterance and the metadata, and locational features; and

apply the statistical classifier model to the utterance.

10. The computing device of claim 8, wherein the processor, in identifying an item among the plurality of items on the display, is operative to:

identify an explicit reference to the item based on a portion of text in the utterance;

identify an explicit reference to the item based on a textual cue in the utterance; and

identify an explicit reference to the item based on screen location data in the utterance.

11. The computing device of claim 10, wherein the processor, in identifying an item among the plurality of items on the display, is further operative to:

identify an implicit reference to the item based on the utterance; and

identify an implicit reference to a position of the item based on the utterance.

12. The computing device of claim 11, wherein the processor, in identifying an item among the plurality of items on the display, is further operative to:

determine a plurality of match scores between the utterance and the metadata;

parse the utterance for matching phrases between semantic phrases in the utterance and the metadata; and

parse the utterance to capture location indicators for predicting a screen location of the item.

13. The computing device of claim 8, wherein the processor, in performing an action associated with the utterance upon identifying the item, is operative to:

select the item on the display; and

display additional information about the selected item.

14. The computing device of claim 8, wherein the processor, in performing an action associated with the utterance upon identifying the item, is operative to:

select the item on the display; and

execute an activity associated with the selected item.

15. A computer-readable storage medium storing computer executable instructions which, when executed by a computer, will cause computer to perform a method comprising:

receiving a first utterance comprising a query for a plurality of items;

displaying the plurality of items in response to the first utterance on a display;

storing metadata related to the plurality of items;

receiving a second utterance;

applying a disambiguation model to the second utterance, wherein the applying includes;

identifying whether the second utterance is directed to at least one item in the plurality of items on the display,

extracting a plurality of referential features from the second utterance, and

identifying an item among the plurality of items on the display corresponding to the second utterance by comparing at least some of the plurality of referential features extracted from the second utterance to the metadata and by determining a lexical match between the utterance and the metadata associated with the item; and

performing, by the computing device, an action associated with the second utterance upon identifying the item corresponding to the second utterance.

16. The computer-readable storage medium of claim 15, wherein identifying whether the second utterance is directed at least one of the plurality of items comprises:

building a statistical classifier model comprising lexical and semantic features, the lexical and semantic features comprising one or more of the following;

a vocabulary obtained from text in the second utterance, a phrase match between the second utterance and the metadata, and locational features; and

applying the statistical classifier model to the second utterance.

17. The computer-readable storage medium of claim 15, wherein identifying an item among the plurality of items comprises:

identifying an explicit reference to the item based on a portion of text in the second utterance;

identifying an explicit reference to the item based on a textual cue in the second utterance; and

identifying an explicit reference to the item based on screen location data in the second utterance.

18. The computer-readable storage medium of claim 17, wherein identifying an item among the plurality of items further comprises:

identifying an implicit reference to the item based on the second utterance; and

identifying an implicit reference to a position of the item based on the second utterance.

19. The computer-readable storage medium of claim 18, wherein identifying an item among the plurality of items further comprises:

determining a plurality of match scores between the second utterance and the metadata;

parsing the second utterance for matching phrases between semantic phrases in the second utterance and the metadata; and

parsing the second utterance to capture location indicators for predicting a screen location of the item.

20. The computer-readable storage medium of claim 15, wherein performing, by the computing device, an action associated with the second utterance, comprises:

selecting the item on the display;

displaying additional information about the selected item; and

executing an activity associated with the selected item.