NAMED ENTITY VARIATIONS FOR MULTIMODAL UNDERSTANDING SYSTEMS

• US 20140180676A1
• Filed: 12/21/2012
• Published: 06/26/2014
• Est. Priority Date: 12/21/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Celikyilmaz, Fethiye Asli, Hakkani-Tur, Dilek, Iyer, Rukmini, Tur, Gokhan, Hillard, Dustin
• Granted Patent: US 9,916,301 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/295 Named entity recognition

Abstract

Click logs are automatically mined to assist in discovering candidate variations for named entities. The named entities may be obtained from one or more sources and include an initial list of named entities. A search may be performed within one or more search engines to determine common phrases that are used to identify the named entity in addition to the named entity initially included in the named entity list. Click logs associated with results of past searches are automatically mined to discover what phrases determined from the searches are candidate variations for the named entity. The candidate variations are scored to assist in determining the variations to include within an understanding model. The variations may also be used when delivering responses and displayed output in the SLU system. For example, instead of using the listed named entity, a popular and/or shortened name may be used by the system.

Claims

1. A method for determining variations for named entities, comprising:

accessing a named entity list comprising a canonical phrase for each entry in the named entity list;

determining candidate variations for the canonical phrase for one or more of the entries in the named entity list using click data;

determining candidate variations to include in a language understanding model; and

adding the determined candidate variations to the understanding model.

2. The method of claim 1, wherein determining the candidate variations for the canonical phrase for the one or more of the entries in the named entity list comprises performing a search using a search engine with the canonical phrase.

3. The method of claim 2, wherein determining the candidate variations comprises accessing a click log of the search engine to determine a scoring for the determined candidate variations.

4. The method of claim 1, further comprising using at least one of the variations added to the understanding model when delivering a response.

5. The method of claim 1, further comprising using at least one of the candidate variations added to the language understanding model in place of the canonical phrase on a display.

6. The method of claim 1, wherein determining the candidate variations for the canonical phrase comprises determining a distribution of clicks between different web sites.

7. The method of claim 1, further comprising determining a set of related sites based on the click data.

8. The method of claim 1, further comprising scoring each of the candidate variations to how related the candidate variation is related to the corresponding named entity within the named entity list.

9. The method of claim 1, wherein determining the candidate variations to include in the language understanding model comprises determining when a score of each of the candidate variations is above a threshold.

10. A computer-readable medium storing computer-executable instructions for determining variations for named entities, comprising:

accessing a named entity list comprising a canonical phrase for each entry in the named entity list;

determining candidate variations for the canonical phrase for one or more of the entries in the named entity list using click data;

generating a score for each of the candidate variations that represents how related the candidate variation is to the corresponding named entity in the named entity list;

determining candidate variations to include in a language understanding model using the scores; and

adding the determined candidate variations to the language understanding model.

11. The computer-readable medium of claim 10, wherein determining the candidate variations comprises performing a search with at least one of the canonical phrases and obtaining results used to determine the candidate variations.

12. The computer-readable medium of claim 10, wherein determining the candidate variations comprises accessing search data to determine a scoring for the candidate variations.

13. The computer-readable medium of claim 10, further comprising at least one of:

using at least one of the variations added to the language understanding model when delivering a response; and

using at least one of the candidate variations added to the language understanding model in place of the canonical phrase on a display.

14. The computer-readable medium of claim 10, wherein determining the candidate variations comprises determining a distribution of clicks between different web sites.

15. The computer-readable medium of claim 10, further comprising determining a set of related sites based on the click data.

16. The computer-readable medium of claim 10, wherein determining the candidate variations to include in the language understanding model comprises determining when a score of each of the candidate variations is above a threshold.

17. A system for determining variations for named entities, comprising:

a processor and memory;

an operating environment executing using the processor;

a display; and

a multimodal manager that is configured to perform actions comprising;

accessing a named entity list comprising a canonical phrase for each entry in the named entity list;

determining candidate variations for the canonical phrase for one or more of the entries in the named entity list using click data;

generating a score for each of the candidate variations that represents how related the candidate variation is to the corresponding named entity in the named entity list;

determining candidate variations to include in a language understanding model using the scores; and

adding the determined candidate variations to the language understanding model.

18. The system of claim 17, wherein determining the candidate variations comprises performing a search with at least one of the canonical phrases and obtaining results used to determine the candidate variations and determining a scoring for the candidate variations.

19. The system of claim 17, further comprising at least one of:

using at least one of the variations added to the language understanding model when delivering a response; and

using at least one of the candidate variations added to the understanding model in place of the canonical phrase on a display.

20. The system of claim 17, wherein determining the candidate variations comprises determining a distribution of clicks between different web sites and determining a set of related sites based on the click data.