Digital assistant extension automatic ranking and selection

• US 10,437,841 B2
• Filed: 10/10/2016
• Issued: 10/08/2019
• Est. Priority Date: 10/10/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Robichaud, Jean-Philippe
• Primary Examiner(s): Cao, Phuong Thao
• Application Number:
  US15/289,765
• Publication Number:
  US 20180101533A1
• Time in Patent Office: 1,093 Days
• Field of Search: 707709
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 51/02 using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 51/214 using selective forwarding

Abstract

Representative embodiments disclose mechanisms to automatically rank and select extensions triggered in a digital assistant. A sample set of extensions are executed against a set of curated queries in order to extract a set of features and/or statistics. The system trains a machine learning model based on the features and/or statistics to rank and select extensions based on their response to a query. New extension incorporated into the system are executed against a second set of curated queries to obtain a set of extracted features and/or statistics which are saved for use at runtime. At runtime, a query phrase received by the system triggers one or more tasks from extensions. Extracted features for the triggered extensions are combined with stored features/statistics and at least a subset of the results presented to the trained ranking and selection model. The model ranks and selects appropriate tasks which are presented to the user.

Claims

1. A method for selecting and ranking service extensions, comprising:

receiving a query from a user or an inference related to the user;

identifying any bots that activate in response to the query or the inference;

ranking the activated bots by performing acts for each activated bot comprising;

identifying a set of features corresponding to the activated bot, at least a portion of the set of features being pre-calculated based on the activated bot processing a set of curated queries;

presenting at least a portion of the set of features to a ranking and response selection model trained via a machine learning process using the set of curated queries;

identifying a rank for the activated bot using the ranking and response selection model and the portion of the set of features; and

placing the activated bot in rank order compared to other activated bots to create a ranked list of bots;

selecting at least one bot from the ranked list of bots; and

presenting an output of the at least one bot to the user.

2. The method of claim 1, further comprising converting the query from one format to another format for processing.

3. The method of claim 1, wherein the set of features comprises at least one of:

features extracted from results of a language understanding model;

features extracted from how well a conversation progress under a hypothesis that a specific response is accepted and presented to the user;

features extracted from statistics collected for models used by tasks activities of the activated bot;

features extracted from a configuration of the activated bot; and

features extracted from an input source.

4. The method of claim 1, wherein the set of features further comprises at least one of:

a correlation metric between a language understanding model of a bot and a built-in language understanding model;

a list of tasks that the bot handles; and

at least one metric describing at least one query that causes the bot to trigger.

5. The method of claim 1, further comprising;

retrieving a set of curated queries;

executing the set of curated queries against a submitted bot;

gathering a set of features characterizing a response by the submitted bot to the set of curated queries; and

storing the set of features.

6. The method of claim 5, further comprising:

extracting from the submitted bot a set of features comprising tasks handled by the submitted bot.

7. The method of claim 5, further comprising;

identifying a set of positive trigger examples or a set of negative trigger examples or both based on the response by the submitted bot to the set curated queries.

8. The method of claim 1, further comprising:

identifying a set of representative bots;

identifying a training set of curated queries; and

using a machine learning process, creating the ranking and response selection model from the set of represented bots and the training set of curated queries.

9. A computing system comprising:

a processor and executable instructions accessible on a machine-readable medium that, when executed, cause the computing system to perform operations comprising;

receiving a query from a user or an inference related to the user;

identifying any bots that activate in response to the query or the inference;

ranking the activated bots by performing operations for each activated bot comprising;

identifying a set of features corresponding to the activated bot, at least a portion of the set of features being pre-calculated based on the activated bot processing a set of c mated queries;

presenting at least a portion of the set of features to a ranking and response selection model trained via a machine learning process using the set of curated queries;

identifying a rank for the activated bot using the ranking and response selection model and the portion of the set of features; and

placing the activated bot in rank order compared to other activated bots to create a ranked list of bots;

selecting at least one bot from the ranked list of bots; and

presenting an output of the at least one bot to the user.

10. The computing system of claim 9, wherein the set of features comprises features corresponding to the activated bot in the context of other activated bots and features calculated from a set of curated queries.

11. The computing system of claim 9, wherein the set of features comprises at least one of;

a confidence score of a language understanding model;

an indicator of entities detected by the language understanding model;

a confidence score distribution;

a frequency of triggering on top of an existing intent; and

an indicator of a type of queries that cause triggering.

12. The computing system of claim 9, wherein the set of features further comprises at least one of:

a correlation metric between a language understanding model of a bot and a built-in language understanding model;

a list of tasks that the bot handles; and

at least one metric describing at least one query that causes the bot to trigger.

13. The computing system of claim 9, further comprising:

retrieving a set of curated queries;

executing the set of curated queries against a submitted bot;

gathering a set of features characterizing a response by the submitted bot to the set of coated queries; and

storing the set of features.

14. The computing system of claim 13, further comprising:

extracting from the submitted bot a set of features comprising tasks handled by the submitted bot.

15. The computing system of claim 13, further comprising:

identifying a set of positive trigger examples or a set of negative trigger examples or both based on the submitted bot response to the set of curated queries.

16. The computing system of claim 9, further comprising:

identifying a set of representative bots;

identifying a training set of curated queries; and

using a machine learning process, creating the ranking and response selection model from the set of represented bots and the training set of curated queries.

17. A non-transitory machine-readable medium having executable instructions encoded thereon, which, when executed by at least one processor of a machine, cause the machine to perform operations comprising:

training a ranking and response selection model by performing operations comprising;

identifying a set of representative bots;

identifying a training set of curated queries; and

using a machine learning process, creating the ranking and response selection

model from the set of represented bots and the training set of curated queries;

receiving a query from a user or an inference related to the user;

identifying any bots that activate in response to the query or the inference;

ranking the activated bots by performing operations for each activated bot comprising;

identifying a set of features corresponding to the activated bot in the context of other activated bots;

identifying a second set of features calculated from a second set of curated queries;

combining the set of features and the second set of features into a set of extracted features;

presenting a subset of the set of extracted features to the ranking and response selection model;

identifying a rank for the activated bot using the ranking and response selection model and the subset of extracted features; and

placing the activated bot in rank order compared to other activated bots to create a ranked list of bots;

selecting at least one bot from the ranked list of bots; and

presenting an output of the at least one bot to the user.

18. The non-transitory machine-readable medium of claim 17, wherein the machine learning process comprises at least one of:

a regression based method;

an instance based method;

a regularization method;

a decision tree method;

a Bayesian method;

a clustering method;

an association rule learning method;

a neural network method;

a deep learning method;

a dimensionality reduction method; and

an ensemble method.

19. The non-transitory machine-readable medium of claim 17, wherein the second set of features is extracted by operations comprising:

retrieving a set of curated queries;

executing the set of curated queries against a submitted bot;

gathering the second set of features characterizing a response by the submitted bot to the set of curated queries; and

storing the set of features.

20. The non-transitory machine-readable medium of claim 17, wherein the set of features or second set of features comprises at least one of:

features extracted from results of a language understanding model;

features extracted from how well a conversation progress under a hypothesis that a specific response is accepted and presented to the user;

features extracted from statistics collected for models used by tasks activities of the activated bot;

features extracted from a configuration of the activated bot; and

features extracted from an input source.