Method and system for crowdsourcing software quality testing and error detection in a tax return preparation system

• US 10,162,734 B1
• Filed: 07/20/2016
• Issued: 12/25/2018
• Est. Priority Date: 07/20/2016
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Podgorny, Igor A., Cessna, Joseph, Gielow, Christopher C., Cannon, Matthew
• Primary Examiner(s): Lottich, Joshua P
• Application Number:
  US15/214,927
• Time in Patent Office: 888 Days
• CPC Class Codes: G06F 11/366 using diagnostics G06F11/0703 takes precedence
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/041 Abduction
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 30/016 After-sales
• CPC Class Codes: G06Q 40/123 Tax preparation or submission
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

Disclosed methods and systems crowdsource quality assurance testing and error detection for user experience pages of a tax return preparation system, according to one embodiment. Testing and debugging combinations and sequences user experience pages can be time consuming and costly, and crowdsourcing error detection can reduce the length of time typically consumed in detecting all errors in a user experience flow, according to one embodiment. The methods and system include monitoring users'"'"' navigation behavior (e.g., system access data) within one or more user experience pages, using predictive models to detect when users are experiencing a potential issue, and providing issue resolution content to both help the users and characterize the potential issue, according to one embodiment. If the potential issue is an error, the methods and systems include requesting more information about the error, to facilitate resolving the error for current and subsequent users, according to one embodiment.

Claims

1. A computing system implemented method for detecting errors in one or more user experience pages of a tax return preparation system, comprising:

providing, with one or more computing systems, a tax return preparation system;

providing, with the tax return preparation system, tax return interview content data representing one or more user experience pages of a tax return preparation interview, to obtain user characteristics data from a user computing system for a user to enable the tax return preparation system to prepare a tax return for the user;

monitoring system access data representing one or more system access activities by the user within one or more of the user experience pages of the tax return preparation interview;

storing the user characteristics data and the system access data of the user in memory allocated for use by the tax return preparation system;

providing predictive model data representing one or more predictive models that are trained to determine a likelihood of a potential issue with one or more of the user experience pages of the tax return preparation interview, at least partially based on one or more of the user characteristics data and the system access data;

applying one or more of the user characteristics data and the system access data to the predictive model data to transform one or more of the user characteristics data and the system access data into potential issue score data representing one or more potential issue scores that represent the likelihood of a potential issue in one or more of the user experience pages;

applying threshold data to the potential issue score data to determine if one or more of the potential issue scores exceed one or more thresholds represented by the threshold data;

if one or more of the potential issue scores exceed one or more of the thresholds, providing issue resolution content data to the user to receive user response data that characterizes the potential issue and that indicates whether the potential issue is a user question related to one or more of the user experience pages or an error in one or more of the user experience pages;

if the user response data indicates that the potential issue is the user question, displaying similar user questions and responses to the similar user questions, to improve a user experience of the user in the tax return preparation system; and

if the user response data indicates that the potential issues is the error, alerting one or more service provider support personnel of the potential issue, to facilitate repair of the error and to improve user experiences of subsequent users of the tax return preparation system.

2. The computing system implemented method of claim 1, wherein the issue resolution content data includes:

first user input element data representing first user input elements that receive issue selection data from the user, the issue selection data representing the error in one or more of the user experience pages or the user question that is related to one or more of the user experience pages,

wherein the user response data includes the issue selection data.

3. The computing system implemented method of claim 2, wherein the issue resolution content data includes:

second user input element data representing a second user input element that receives user question data from the user, the user question data representing the user question that related to one or more of the user experience pages,

wherein the user response data includes the user question data.

4. The computing system implemented method of claim 3, wherein the first user input elements are user interface buttons, and the second user input element is a input text box.

5. The computing system implemented method of claim 1, wherein the issue resolution content data includes question improvement suggestions that are at least partially based on a linguistic analysis of search terms of the user question.

6. The computing system implemented method of claim 5, further comprising:

providing an issue resolution engine that generates the question improvement suggestions by performing the linguistic analysis of the search terms of the user question.

7. The computing system implemented method of claim 5, wherein the question improvement suggestions increase a likelihood that an online community will return a user question response that is satisfactory to the user, wherein the online community crowdsources responses to user questions arising from use of the tax return preparation system.

8. The computing system implemented method of claim 1, wherein the system access data includes user system characteristics, the user system characteristics including one or more of:

an operating system used by the user computing system to access the tax return preparation system;

a hardware identifier of the user computing system used to access the tax return preparation system; and

a web browser used by the user computing system to access the tax return preparation system.

9. The computing system implemented method of claim 8, wherein the system access data is at least partially based on clickstream data from the user computing system.

10. The computing system implemented method of claim 1, further comprising:

generating receiver operating characteristics data representing receiver operating characteristics of one or more of the predictive models; and

determining one or more of the thresholds at least partially based on the receiver operating characteristics of one or more of the predictive models.

11. The computing system implemented method of claim 1, wherein the system access data is selected from a group of system access data consisting of:

data representing features or characteristics associated with an interaction between the user computing system and the tax return preparation system;

data representing a web browser of the user computing system;

data representing an operating system of the user computing system;

data representing a media access control address of the user computing system;

data representing user credentials used to access the tax return preparation system;

data representing a user account;

data representing a user account identifier;

data representing how quickly user credentials are provided to the tax return preparation system to obtain access to the user account;

data representing a duration that a user computing system spends on a particular user experience page;

data representing a user flipping back and forth between two of more of the user experience pages;

data representing a selection tool repeatedly hovering over a help menu or help icon;

data representing comparisons between navigation behavior for the user and navigation behavior of other users within the tax return preparation system;

data representing a duration of an access session between the user computing system and the tax return preparation system;

data representing a duration for preparing a tax return for the user;

data representing conflicts between visited user experience pages and user experience pages that the user is highly likely to have visited at least partially based on the user characteristics data; and

data representing an IP address of the user computing system.

12. The computing system implemented method of claim 1, wherein displaying similar user questions and responses to the similar user questions, includes:

analyzing search terms of the user question;

determining one or more topics for the user question;

searching a user question database for the similar user questions at least partially based on the one or more topics for the user question, wherein the similar user questions include at least some of the one or more topics for the user question;

retrieving hyperlinks or other web-based references to the similar user questions; and

displaying the hyperlinks or other web-based references to the similar user questions to enable a user to access the similar user questions.

13. The computing system implemented method of claim 1, further comprising:

training the predictive model data by applying a predictive model training operation to prior system access data and/or prior user characteristics data, the prior system access data representing system access activities of prior users of the tax return preparation system, the prior user characteristics data representing user characteristics of the prior users of the tax return preparation system.

14. The computing system implemented method of claim 13, wherein, the predictive model training operation is selected from a group of predictive model training operations, consisting of:

regression;

logistic regression;

decision trees;

artificial neural networks;

support vector machines;

linear regression;

nearest neighbor methods;

distance based methods;

naive Bayes;

linear discriminant analysis; and

k-nearest neighbor algorithm.

15. The computing system implemented method of claim 1, wherein the user characteristics data is selected from a group of user characteristics data, consisting of:

data indicating an age of the user;

data indicating an age of a spouse of the user;

data indicating a zip code;

data indicating a tax return filing status;

data indicating state income;

data indicating a home ownership status;

data indicating a home rental status;

data indicating a retirement status;

data indicating a student status;

data indicating an occupation of the user;

data indicating an occupation of a spouse of the user;

data indicating whether the user is claimed as a dependent;

data indicating whether a spouse of the user is claimed as a dependent;

data indicating whether another taxpayer is capable of claiming the user as a dependent;

data indicating whether a spouse of the user is capable of being claimed as a dependent;

data indicating salary and wages;

data indicating taxable interest income;

data indicating ordinary dividend income;

data indicating qualified dividend income;

data indicating business income;

data indicating farm income;

data indicating capital gains income;

data indicating taxable pension income;

data indicating pension income amount;

data indicating IRA distributions;

data indicating unemployment compensation;

data indicating taxable IRA;

data indicating taxable Social Security income;

data indicating amount of Social Security income;

data indicating amount of local state taxes paid;

data indicating whether the user filed a previous years'"'"' federal itemized deduction;

data indicating whether the user filed a previous years'"'"' state itemized deduction; and

data indicating whether the user is a returning user to a tax return preparation system;

data indicating an annual income;

data indicating an employer'"'"'s address;

data indicating contractor income;

data indicating a marital status;

data indicating a medical history;

data indicating dependents;

data indicating assets;

data indicating spousal information;

data indicating children'"'"'s information;

data indicating an address;

data indicating a name;

data indicating a date of birth;

data indicating educator expenses;

data indicating health savings account deductions;

data indicating moving expenses;

data indicating IRA deductions;

data indicating student loan interest deductions;

data indicating tuition and fees;

data indicating medical and dental expenses;

data indicating state and local taxes;

data indicating real estate taxes;

data indicating personal property tax;

data indicating mortgage interest;

data indicating charitable contributions;

data indicating casualty and theft losses;

data indicating unreimbursed employee expenses;

data indicating an alternative minimum tax;

data indicating a foreign tax credit;

data indicating education tax credits;

data indicating retirement savings contributions; and

data indicating child tax credits.

16. A computing system implemented method for detecting errors in one or more user experience pages of a tax return preparation system, comprising:

providing, with one or more computing systems, a tax return preparation system;

providing, with the tax return preparation system, tax return interview content data representing one or more user experience pages of a tax return preparation interview, to obtain user characteristics data from a user computing system for a user to enable the tax return preparation system to prepare a tax return for the user;

receiving system access data representing one or more system access activities by the user within one or more of the user experience pages of the tax return preparation interview;

storing the user characteristics data and the system access data of the user in memory allocated for use by the tax return preparation system;

providing predictive model data representing one or more predictive models that are trained to determine a likelihood of a potential issue with one or more of the user experience pages of the tax return preparation interview, at least partially based on the system access data;

applying the system access data to the predictive model data to generate potential issue score data at least partially based on the system access data, the predictive model data representing one or more potential issue scores that represent the likelihood of a potential issue in one or more of the user experience pages;

providing issue resolution content data to the user to notify the user that the tax return preparation system detects a potential issue for the user within the tax return preparation interview, at least partially based on one or more of the potential issue scores;

receiving user response data from the user in response to providing the issue resolution data to the user, the user response data representing a user response that characterizes the potential issue as a user question with content of one or more of the user experience pages or as an error in one or more of the user experience pages;

if the user response data indicates that the potential issue is the user question, displaying similar user questions and responses to the similar user questions, to improve a user experience of the user in the tax return preparation system; and

if the user response data indicates that the potential issue is the error, alerting one or more service provider support personnel of the potential issue, to facilitate repair of the error and to improve user experiences of subsequent users of the tax return preparation system.

17. The computing system implemented method of claim 16, wherein the issue resolution content data includes:

first user input element data representing first user input elements that receive issue selection data from the user, the issue selection data representing the error in one or more of the user experience pages or the user question that is related to one or more of the user experience pages; and

second user input element data representing a second user input element that receives user question data from the user, the user question data representing the user question that related to one or more of the user experience pages,

wherein the user response data includes the issue selection data and the question data.

18. The computing system implemented method of claim 16, wherein the issue resolution content data includes question improvement suggestions that are at least partially based on a linguistic analysis of search terms of the user question.

19. The computing system implemented method of claim 18, further comprising:

providing an issue resolution engine that generates the question improvement suggestions by performing the linguistic analysis of the search terms of the user question.

20. The computing system implemented method of claim 18, wherein the question improvement suggestions increase a likelihood that an online community will return a user question response that is satisfactory to the user, wherein the online community crowdsources responses to user questions arising from use of the tax return preparation system.

21. The computing system implemented method of claim 16, wherein the system access data is selected from a group of system access data consisting of:

data representing features or characteristics associated with an interaction between the user computing system and the tax return preparation system;

data representing a web browser of the user computing system;

data representing an operating system of the user computing system;

data representing a media access control address of the user computing system;

data representing user credentials used to access the tax return preparation system;

data representing a user account;

data representing a user account identifier;

data representing how quickly user credentials are provided to the tax return preparation system to obtain access to the user account;

data representing a duration that a user computing system spends on a particular user experience page;

data representing a user flipping back and forth between two of more of the user experience pages;

data representing a selection tool repeatedly hovering over a help menu or help icon;

data representing comparisons between navigation behavior for the user and navigation behavior of other users within the tax return preparation system;

data representing a duration of an access session between the user computing system and the tax return preparation system;

data representing a duration for preparing a tax return for the user;

data representing conflicts between visited user experience pages and user experience pages that the user is highly likely to have visited at least partially based on the user characteristics data; and

data representing an IP address of the user computing system.

22. The computing system implemented method of claim 16, wherein displaying similar user questions and responses to the similar user questions, includes:

analyzing search terms of the user question;

determining one or more topics for the user question;

searching a user question database for the similar user questions at least partially based on the one or more topics for the user question, wherein the similar user questions include at least some of the one or more topics for the user question;

retrieving hyperlinks or other web-based references to the similar user questions; and

displaying the hyperlinks or other web-based references to the similar user questions to enable a user to access the similar user questions.

23. The computing system implemented method of claim 16, further comprising:

training the predictive model data by applying a predictive model training operation to prior system access data and/or prior user characteristics data, the prior system access data representing system access activities of prior users of the tax return preparation system, the prior user characteristics data representing user characteristics of the prior users of the tax return preparation system.

24. The computing system implemented method of claim 16, further comprising:

applying user characteristics data of the user to the predictive model to generate one or more of the potential issue scores, wherein the user characteristics data is selected from a group of user characteristics data, consisting of;

data indicating an age of the user;

data indicating an age of a spouse of the user;

data indicating a zip code;

data indicating a tax return filing status;

data indicating state income;

data indicating a home ownership status;

data indicating a home rental status;

data indicating a retirement status;

data indicating a student status;

data indicating an occupation of the user;

data indicating an occupation of a spouse of the user;

data indicating whether the user is claimed as a dependent;

data indicating whether a spouse of the user is claimed as a dependent;

data indicating whether another taxpayer is capable of claiming the user as a dependent;

data indicating whether a spouse of the user is capable of being claimed as a dependent;

data indicating salary and wages;

data indicating taxable interest income;

data indicating ordinary dividend income;

data indicating qualified dividend income;

data indicating business income;

data indicating farm income;

data indicating capital gains income;

data indicating taxable pension income;

data indicating pension income amount;

data indicating IRA distributions;

data indicating unemployment compensation;

data indicating taxable IRA;

data indicating taxable Social Security income;

data indicating amount of Social Security income;

data indicating amount of local state taxes paid;

data indicating whether the user filed a previous years'"'"' federal itemized deduction;

data indicating whether the user filed a previous years'"'"' state itemized deduction; and

data indicating whether the user is a returning user to a tax return preparation system;

data indicating an annual income;

data indicating an employer'"'"'s address;

data indicating contractor income;

data indicating a marital status;

data indicating a medical history;

data indicating dependents;

data indicating assets;

data indicating spousal information;

data indicating children'"'"'s information;

data indicating an address;

data indicating a name;

data indicating a date of birth;

data indicating educator expenses;

data indicating health savings account deductions;

data indicating moving expenses;

data indicating IRA deductions;

data indicating student loan interest deductions;

data indicating tuition and fees;

data indicating medical and dental expenses;

data indicating state and local taxes;

data indicating real estate taxes;

data indicating personal property tax;

data indicating mortgage interest;

data indicating charitable contributions;

data indicating casualty and theft losses;

data indicating unreimbursed employee expenses;

data indicating an alternative minimum tax;

data indicating a foreign tax credit;

data indicating education tax credits;

data indicating retirement savings contributions; and

data indicating child tax credits.

25. A system that detects errors in one or more user experience pages of a tax return preparation system, comprising:

one or more processors; and

memory having instructions which, if executed by the one or more processors, cause the one or more processors to perform a process for detecting errors in one or more user experience pages of a tax return preparation system, the process comprising;

providing the tax return preparation system;

providing, with the tax return preparation system, tax return interview content data representing one or more of the user experience pages of a tax return preparation interview, to obtain user characteristics data from a user computing system for a user to enable the tax return preparation system to prepare a tax return for the user;

receiving system access data representing one or more system access activities by the user within one or more of the user experience pages of the tax return preparation interview;

storing the user characteristics data and the system access data of the user in memory allocated for use by the tax return preparation system;

providing predictive model data representing one or more predictive models that are trained to determine a likelihood of a potential issue with one or more of the user experience pages of the tax return preparation interview, at least partially based on the system access data;

applying the system access data to the predictive model data to generate potential issue score data at least partially based on the system access data, the predictive model data representing one or more potential issue scores that represent the likelihood of a potential issue in one or more of the user experience pages;

providing issue resolution content data to the user to notify the user that the tax return preparation system detects a potential issue for the user within the tax return preparation interview, at least partially based on one or more of the potential issue scores;

receiving user response data from the user in response to providing the issue resolution data to the user, the user response data representing a user response that characterizes the potential issue as a user question with content of one or more of the user experience pages or as an error in one or more of the user experience pages;

if the user response data indicates that the potential issue is the user question, displaying similar user questions and responses to the similar user questions, to improve a user experience of the user in the tax return preparation system; and

if the user response data indicates that the potential issue is the error, alerting one or more service provider support personnel of the potential issue, to facilitate repair of the error and to improve user experiences of subsequent users of the tax return preparation system.

26. The system of claim 25, wherein the issue resolution content data includes:

first user input element data representing first user input elements that receive issue selection data from the user, the issue selection data representing the error in one or more of the user experience pages or the user question that is related to one or more of the user experience pages; and

second user input element data representing a second user input element that receives user question data from the user, the user question data representing the user question that related to one or more of the user experience pages,

wherein the user response data includes the issue selection data and the question data.

27. The system of claim 25, wherein the issue resolution content data includes question improvement suggestions that are at least partially based on a linguistic analysis of search terms of the user question.

28. The system claim 27, wherein the question improvement suggestions increase a likelihood that an online community will return a user question response that is satisfactory to the user, wherein the online community crowdsources responses to user questions arising from use of the tax return preparation system.

29. The system claim 25, wherein the system access data is selected from a group of system access data consisting of:

data representing features or characteristics associated with an interaction between the user computing system and the tax return preparation system;

data representing a web browser of the user computing system;

data representing an operating system of the user computing system;

data representing a media access control address of the user computing system;

data representing user credentials used to access the tax return preparation system;

data representing a user account;

data representing a user account identifier;

data representing how quickly user credentials are provided to the tax return preparation system to obtain access to the user account;

data representing a duration that a user computing system spends on a particular user experience page;

data representing a user flipping back and forth between two of more of the user experience pages;

data representing a selection tool repeatedly hovering over a help menu or help icon;

data representing comparisons between navigation behavior for the user and navigation behavior of other users within the tax return preparation system;

data representing a duration of an access session between the user computing system and the tax return preparation system;

data representing a duration for preparing a tax return for the user;

data representing conflicts between visited user experience pages and user experience pages that the user is highly likely to have visited at least partially based on the user characteristics data; and

data representing an IP address of the user computing system.

30. The system of claim 25, wherein displaying similar user questions and responses to the similar user questions, includes:

analyzing search terms of the user question;

determining one or more topics for the user question;

searching a user question database for the similar user questions at least partially based on the one or more topics for the user question, wherein the similar user questions include at least some of the one or more topics for the user question;

retrieving hyperlinks or other web-based references to the similar user questions; and

displaying the hyperlinks or other web-based references to the similar user questions to enable a user to access the similar user questions.