Method and system for improving content searching in a question and answer customer support system by using a crowd-machine learning hybrid predictive model

• US 10,467,541 B2
• Filed: 07/27/2016
• Issued: 11/05/2019
• Est. Priority Date: 07/27/2016
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Podgorny, Igor A., Indyk, Benjamin, Cannon, Matthew, Guidry, Jonathan, Ma, Irene F.
• Primary Examiner(s): Cole, Brandon S
• Application Number:
  US15/220,677
• Publication Number:
  US 20180032890A1
• Time in Patent Office: 1,196 Days
• Field of Search: 706 1- 62
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 30/016 After-sales

Abstract

A method and system improves content searching in a question and answer customer support system by using a crowd-machine learning hybrid predictive model, according to one embodiment. The question and answer customer support system determines which customer support content to provide to users by using the hybrid predictive model, according to one embodiment. The question and answer customer support system receives a search query from a user and applies the search query (or a representation of the search query) to the hybrid predictive model, according to one embodiment. The hybrid predictive model generates a likelihood that particular customer support content is relevant to a user'"'"'s search query, according to one embodiment. The question and answer customer support system acquires user feedback from users and updates/trains the hybrid predictive model based on the user feedback, according to one embodiment.

Claims

1. A computing system implemented method for using a hybrid predictive model to respond to search queries for customer support content in a question and answer customer support system, the method comprising:

storing existing customer support content data in memory allocated for use by a question and answer customer support system, the existing customer support content data representing existing customer support content entries having groups of combinations of existing search queries and existing responses, the existing search queries having been submitted to the question and answer customer support system by prior users and the existing responses having been submitted to the question and answer customer support system in response to the existing search queries;

receiving search query data from a user, the search query data representing a current search query for customer support content from the question and answer customer support system;

training a hybrid predictive model through unsupervised learning using at least a combination of machine learning content and crowdsourced user feedback to identify the customer support content of the current search query, resulting in hybrid predictive model data, the training at least including applying one or more additional natural language processing algorithms to the crowdsourced user feedback to extract one or more features of the customer support content from the user feedback, the training including consideration of a length of reference material and a style of a reference material;

applying the search query data to the predictive model data to generate content selection score data representing two or more content selection scores, at least one of the two or more content selection scores representing a relevance of the existing customer support content entries to the search query data, and at least one of the two or more content selection scores representing a quality of the customer support content, the quality-related content selection score being provided by a content quality predictive model;

selecting one of the existing customer support content entries at least partially based on the one or more content selection scores;

generating user experience display data that includes at least one user experience page and that includes the one of the existing customer support entries, the user experience display data representing a user experience display that renders the one of the existing customer support content entries in the at least one user experience page;

providing the user experience display data to the user, in response to receiving the search query data from the user;

requesting user feedback data from the user, to enable the question and answer customer support system to improve performance of the hybrid predictive model, the user feedback data representing crowdsourced user feedback related to the one of the existing customer support content entries that is included in the user experience display data; and

updating, upon receipt of user feedback data from the user, the hybrid predictive model at least partially based on the user feedback data, the received feedback including data regarding a quality of writing, accuracy and content and search ranking.

2. The computing system implemented method of claim 1, wherein the combination of machine learning content includes features of the existing customer support content data that are identified using one or more machine learning techniques.

3. The computing system implemented method of claim 1, wherein the crowdsourced user feedback includes features of the existing customer support content data that are extracted from the crowdsourced user feedback and/or from the user feedback data.

4. The computing system implemented method of claim 1, further comprising:

generating the hybrid predictive model data by applying one or more predictive model training operations to;

features of the existing customer support content data that are identified by applying one or more first machine learning techniques to the existing customer support content data;

features of the existing customer support content data that are identified by applying one or more second machine learning techniques to the crowdsourced user feedback and/or to the user feedback data; and

the existing customer support content data.

5. The computing system implemented method of claim 4, wherein the one or more predictive model training operations are selected from a group of predictive model training operations, consisting of:

regression;

logistic regression;

decision trees;

artificial neural networks;

support vector machines;

linear regression;

nearest neighbor methods;

distance based methods;

naive Bayes;

linear discriminant analysis; and

k-nearest neighbor algorithm.

6. The computing system implemented method of claim 4, wherein features of the existing customer support content data that are identified by applying one or more first machine learning techniques to the existing customer support content data, or features of the existing customer support content data that are identified by applying one or more second machine learning techniques to the crowdsourced user feedback and/or to the user feedback data, are selected from a group of features, consisting of:

length of a search query;

length of a response;

tone and/or sentiment of a search query;

tone and/or sentiment of a response;

topic of a search query;

topic of a response;

quality of writing of a search query;

quality of writing of a response;

relevance of a search query to a user'"'"'s search query;

relevance of response to a search query;

relevance of a response to a user'"'"'s search query;

punctuation of a search query;

punctuation of a response;

quality of grammar of a search query or response;

perceived accuracy of a response;

a search ranking of a combination of a search query and response;

a number of responses associated with a search query;

an age of a search query;

an age of response;

a length of reference material;

a style of a reference material;

an author of a search query;

an author of a response;

an author of a reference material;

attributes of an author of a search query and/or response and/or reference material;

whether a customer support content includes a hyperlink;

a text size of a search query and/or response and/or reference material;

background and/or foreground colors and/or images used to present customer support content;

font characteristics of a customer support content;

whether a details and/or context are provided with a search query;

whether a user felt like a customer support content was helpful;

a number of other users who found a customer support content helpful;

a number of customers who viewed a customer support content;

a type of reference material; and

a number of similar questions displayed with a customer support content.

7. The computing system implemented method of claim 1, wherein the existing customer support content entries include customer support reference materials including one or more customer support articles, customer support dictionaries, and customer support self-help guides.

8. The computing system implemented method of claim 1, wherein the combination of machine learning content includes features of the existing customer support content data that are identified using a probabilistic topic model that is at least partially based on a Latent Dirichlet allocation algorithm.

9. The computing system implemented method of claim 1, wherein requesting the user feedback data includes displaying feedback acquisition features in the at least one user experience page, the feedback acquisition features including one or more of:

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was useful;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was useful;

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was accurate;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was accurate;

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was relevant;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was relevant;

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was well-written;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was well-written;

structured feedback user experience elements configured to determine which topic the user thought the one of the existing customer support content entries was related to; and

unstructured feedback user experience elements configured to determine which topic the user thought the one of the existing customer support content entries was related to.

10. The computing system implemented method of claim 9, wherein at least part of the feedback acquisition features are selected using a feedback acquisition predictive model that selects user experience elements at least partially based on the search query data and/or user characteristics of the user.

11. A computing system implemented method for using a hybrid predictive model to respond to search queries for customer support content in a question and answer customer support system, the method comprising:

storing existing customer support content data in memory allocated for use by a question and answer customer support system, the existing customer support content data representing existing customer support content entries, the existing customer support content entries including groups of combinations of existing search queries and existing responses, the existing search queries having been submitted to the question and answer customer support system by prior users and the existing responses having been submitted to the question and answer customer support system in response to the existing search queries;

receiving search query data from a user, the search query data representing a current search query for customer support content from the question and answer customer support system;

training a hybrid predictive model through unsupervised learning using at least a combination of machine learning content and crowdsourced user feedback to identify the customer support content of the current search query, resulting in hybrid predictive model data, the training at least including applying one or more additional natural language processing algorithms to the crowdsourced user feedback to extract one or more features of the customer support content from the user feedback, the training including consideration of a length of reference material and a style of a reference material;

applying the search query data to the predictive model data to generate content selection score data representing two or more content selection scores that represent a likelihood of satisfaction of the user with the existing customer support entries, at least one of the two or more content selection scores representing a quality of the customer support content, the quality-related content selection score being provided by a content quality predictive model;

selecting one of the existing customer support content entries at least partially based on the one or more content selection scores;

generating user experience display data that includes at least one user experience page and that includes the one of the existing customer support entries, the user experience display data representing a user experience display that renders the one of the existing customer support content entries in the at least one user experience page;

providing the user experience display data to the user, in response to receiving the search query data from the user;

requesting user feedback data from the user, to enable the question and answer customer support system to improve performance of the hybrid predictive model, the user feedback data representing crowdsourced user feedback related to the one of the existing customer support content entries that is included in the user experience display data; and

updating, upon receipt of user feedback data from the user, the hybrid predictive model at least partially based on the user feedback data, the received feedback including data regarding a quality of writing, perceived accuracy or content and search ranking.

12. The computing system implemented method of claim 11, further comprising:

generating the hybrid predictive model data by applying one or more predictive model training operations to;

the features being determined at least partially based on crowdsourced user feedback to the existing customer support content entries; and

the existing customer support content data.

13. The computing system implemented method of claim 11, wherein the features are selected from a group of features, consisting of:

length of a search query;

length of a response;

tone and/or sentiment of a search query;

tone and/or sentiment of a response;

topic of a search query;

topic of a response;

quality of writing of a search query;

quality of writing of a response;

relevance of a search query to a user'"'"'s search query;

relevance of response to a search query;

relevance of a response to a user'"'"'s search query;

punctuation of a search query;

punctuation of a response;

quality of grammar of a search query or response;

perceived accuracy of a response;

a search ranking of a combination of a search query and response;

a number of responses associated with a search query;

an age of a search query;

an age of response;

a length of reference material;

a style of a reference material;

an author of a search query;

an author of a response;

an author of a reference material;

attributes of an author of a search query and/or response and/or reference material;

whether a customer support content includes a hyperlink;

a text size of a search query and/or response and/or reference material;

background and/or foreground colors and/or images used to present customer support content;

font characteristics of a customer support content;

whether a details and/or context are provided with a search query;

whether a user felt like a customer support content was helpful;

a number of other users who found a customer support content helpful;

a number of customers who viewed a customer support content; and

a number of similar questions displayed with a customer support content.

14. The computing system implemented method of claim 11, wherein requesting the user feedback data includes displaying feedback acquisition features in the at least one user experience page, the feedback acquisition features including one or more of:

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was useful;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was useful;

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was accurate;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was accurate;

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was relevant;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was relevant;

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was well-written;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was well-written;

structured feedback user experience elements configured to determine which topic the user thought the one of the existing customer support content entries was related to; and

unstructured feedback user experience elements configured to determine which topic the user thought the one of the existing customer support content entries was related to.

15. The computing system implemented method of claim 14, wherein at least part of the feedback acquisition features are selected using a feedback acquisition predictive model that selects user experience elements at least partially based on the search query data and/or user characteristics of the user.

16. The computing system implemented method of claim 11, wherein the features are determined at least partially based on applying machine learning techniques to the existing customer support content entries.

17. A system for using a hybrid predictive model to respond to search queries for customer support content in a question and answer customer support system, the system comprising:

at least one processor; and

at least one memory coupled to the at least one processor, the at least one memory having stored therein instructions which, when executed by any set of the one or more processors, perform a process for using a hybrid predictive model to respond to search queries for customer support content in a question and answer customer support system, the process including;

storing existing customer support content data representing existing customer support content entries, the existing customer support content entries including groups of combinations of existing search queries and existing responses, the existing search queries having been submitted to the question and answer customer support system by prior users and the existing responses having been submitted to a question and answer customer support system in response to the existing search queries;

receiving search query data from a user, the search query data representing a current search query for customer support content from the question and answer customer support system;

training a hybrid predictive model through unsupervised learning using at least a combination of machine learning content and crowdsourced user feedback to identify the customer support content of the current search query, resulting in hybrid predictive model data, the training at least including applying one or more additional natural language processing algorithms to the crowdsourced user feedback to extract one or more features of the customer support content from the user feedback, the training including consideration of a length of reference material and a style of a reference material;

applying the search query data to the predictive model data to generate content selection score data representing two or more content selection scores, at least one of the two or more content selection scores representing a relevance of the existing customer support content entries to the search query data, at least one of the two or more content selection scores representing a quality of the customer support content, the quality-related content selection score being provided by a content quality predictive model;

selecting one of the existing customer support content entries at least partially based on the one or more content selection scores;

generating user experience display data that includes at least one user experience page and that includes the one of the existing customer support entries, the user experience display data representing a user experience display that renders the one of the existing customer support content entries in the at least one user experience page;

providing the user experience display data to the user, in response to receiving the search query data from the user;

requesting user feedback data from the user, to enable the question and answer customer support system to improve performance of the hybrid predictive model, the user feedback data representing crowdsourced user feedback related to the one of the existing customer support content entries that is included in the user experience display data; and

updating, upon receipt of user feedback data from the user, the hybrid predictive model at least partially based on the user feedback data, the received feedback including data regarding a quality of writing, perceived accuracy or content and search ranking.

18. The system of claim 17, wherein the combination of machine learning content includes features of the existing customer support content data that are identified using one or more machine learning techniques.

19. The system of claim 17, wherein the crowdsourced user feedback includes features of the existing customer support content data that are extracted from the crowdsourced user feedback and/or from the user feedback data.

20. The system of claim 17, wherein the process further comprises:

generating the hybrid predictive model data by applying one or more predictive model training operations to;

features of the existing customer support content data that are identified by applying one or more first machine learning techniques to the existing customer support content data;

features of the existing customer support content data that are identified by applying one or more second machine learning techniques to the crowdsourced user feedback and/or to the user feedback data; and

the existing customer support content data.

21. The system of claim 20, wherein features of the existing customer support content data that are identified by applying one or more first machine learning techniques to the existing customer support content data, or features of the existing customer support content data that are identified by applying one or more second machine learning techniques to the crowdsourced user feedback and/or to the user feedback data, are selected from a group of features, consisting of:

length of a search query;

length of a response;

tone and/or sentiment of a search query;

tone and/or sentiment of a response;

topic of a search query;

topic of a response;

quality of writing of a search query;

quality of writing of a response;

relevance of a search query to a user'"'"'s search query;

relevance of response to a search query;

relevance of a response to a user'"'"'s search query;

punctuation of a search query;

punctuation of a response;

quality of grammar of a search query or response;

perceived accuracy of a response;

a search ranking of a combination of a search query and response;

a number of responses associated with a search query;

an age of a search query;

an age of response;

a length of reference material;

a style of a reference material;

an author of a search query;

an author of a response;

an author of a reference material;

attributes of an author of a search query and/or response and/or reference material;

whether a customer support content includes a hyperlink;

a text size of a search query and/or response and/or reference material;

background and/or foreground colors and/or images used to present customer support content;

font characteristics of a customer support content;

whether a details and/or context are provided with a search query;

whether a user felt like a customer support content was helpful;

a number of other users who found a customer support content helpful;

a number of customers who viewed a customer support content; and

a number of similar questions displayed with a customer support content.

22. The system of claim 17, wherein requesting the user feedback data includes displaying feedback acquisition features in the at least one user experience page, the feedback acquisition features including one or more of:

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was useful;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was useful;

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was accurate;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was accurate;

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was relevant;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was relevant;

structured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was well-written;

unstructured feedback user experience elements configured to determine if the user thought the one of the existing customer support content entries was well-written;

structured feedback user experience elements configured to determine which topic the user thought the one of the existing customer support content entries was related to; and

unstructured feedback user experience elements configured to determine which topic the user thought the one of the existing customer support content entries was related to.

23. The system of claim 22, wherein at least part of the feedback acquisition features are selected using a feedback acquisition predictive model that selects user experience elements at least partially based on the search query data and/or user characteristics of the user.