System and method for identifying a subset of total historical users of a document preparation system to represent a full set of test scenarios based on code coverage

• US 10,725,896 B2
• Filed: 12/27/2017
• Issued: 07/28/2020
• Est. Priority Date: 07/15/2016
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Unsal, Cem, Hanekamp, Jr., David A., Joseph, Saneesh, Atkinson, Steven, Artamonov, Michael A.
• Primary Examiner(s): Tapp, Amelia L
• Application Number:
  US15/855,773
• Publication Number:
  US 20180121337A1
• Time in Patent Office: 944 Days
• CPC Class Codes: G06F 11/364 tracing values on a bus
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06Q 40/123 Tax preparation or submission

Abstract

A method and system generate sample data set for efficiently and accurately testing a new calculation for preparing a portion of an electronic document for users of an electronic document preparation system. The method and system receive the new calculation and gather historical use data related to previously prepared electronic documents for a large number of historical users. The method and system group the historical users into groups based on which sections of a previous version of electronic document preparation software were executed for each historical user in preparing electronic documents for the historical users. The groups are then sampled by selecting a small number of historical users from each group.

Claims

1. A system for generating efficient training sets for testing new processes for preparing electronic documents for users of an electronic document preparation system, the system comprising:

at least one processor; and

at least one memory coupled to the at least one processor, the at least one memory having stored therein instructions which, when executed by any set of the one or more processors, perform a process including;

receiving new calculation data related to a new calculation for generating data values for preparing electronic documents for users of an electronic document preparation system;

retrieving historical user data related to electronic documents previously prepared for a plurality of historical users;

executing, for each of the historical users, prior code data corresponding to a previous version of the new calculation;

generating, for each historical user, trace log data indicating sections of the prior code data that were executed for the historical users based on attributes of the historical user;

generating grouping data including a plurality of groups of historical users based on sections of the prior code data executed for the historical users; and

generating training set data by sampling, from each group in the grouping data, one or more historical users.

2. The system of claim 1, wherein the training set data includes the historical user data associated with the historical users sampled from the grouping data.

3. The system of claim 2, wherein the process further includes testing the new calculation data by executing the new calculation data for the training set data.

4. The system of claim 3, wherein the process further includes:

generating results data indicating results of testing the new calculation data; and

outputting the results data.

5. The system of claim 1, wherein the process includes generating the grouping data based on the sections of the prior code data executed for historical users in accordance with the trace log data.

6. The system of claim 5, wherein the process includes generating the grouping data by including a group for each unique combination of executed sections of the prior code data indicated in the trace log data.

7. The system of claim 1, wherein the process includes generating the trace log data with an instrumented run-time engine of a prior code execution and analysis module.

8. The system of claim 1, wherein the process further includes:

generating code coverage data based on the trace log data, the code coverage data indicating a portion of the prior code executed for the historical users; and

generating the grouping data based on the code coverage data.

9. The system of claim 1, wherein the electronic document preparation system is a tax return preparation system.

10. The system of claim 9, wherein the historical user data includes historical user tax related data, and wherein the previously prepared electronic documents are previously prepared tax returns.

11. The system of claim 10, wherein the calculation data includes a calculation for a tax related form associated with a tax return.

12. A method performed by one or more processors of a system, the method comprising:

receiving, from a database, a new calculation for preparing electronic documents for users of an electronic document preparation system;

requesting code coverage data indicating sections of software instructions executed for a plurality of historical users of the electronic document preparation system;

generating groups of historical users based on the sections of software instructions; and

generating training set data by sampling, from each group, one or more historical users.

13. The method of claim 12, wherein the training set data includes historical user data associated with the historical users selected in the sampling data.

14. The method of claim 12, further comprising receiving query data requesting the training set data for testing the new calculation, wherein requesting the code coverage data includes requesting the code coverage data in response to receiving the query data.

15. The method of claim 14, wherein requesting the code coverage data includes requesting the code coverage data from a third-party data services provider.

16. The method of claim 15, wherein requesting the code coverage data includes requesting a portion of the code coverage data based on the query data.

17. The method of claim 16, further comprising providing the code coverage data to the third-party data services provider prior to receiving the query data.

18. The method of claim 14, wherein requesting the code coverage data includes requesting the code coverage data from an internal database of the electronic document preparation system.

19. A method for generating efficient training sets for testing new processes for preparing electronic documents for users of an electronic document preparation system, the method comprising:

receiving new calculation data related to a new calculation for generating data values for preparing electronic documents for users of an electronic document preparation system;

retrieving historical user data including electronic documents previously prepared for a plurality of historical users;

executing software instructions for preparing electronic documents with the historical user data;

generating, for each historical user, trace log data indicating sections of the software instructions that were executed for the historical users;

generating grouping data including a plurality of groups of historical users based on the sections of the software instructions executed for the historical users; and

generating training set data by sampling, from each group in the grouping data, one or more historical users.

20. The method of claim 19, wherein the software instructions include a previous calculation corresponding to the new calculation.

21. The method of claim 19, further comprising generating code coverage data based on the trace log data, the code coverage data identifying the sections of the software code that were executed for each historical user.

22. The method of claim 21, wherein the code coverage data includes a table indicating the sections of the software instructions that were executed for a plurality of historical users.

23. The method of claim 22, wherein the table includes a matrix with rows corresponding to sections of the software instructions, columns corresponding to historical users, and data values indicating whether a software section was executed for a historical user.

24. The method of claim 23, wherein the matrix represents each historical user as a column vector with data values indicating which sections of the software instructions were executed for the historical user.

25. The method of claim 24, further comprising generating the grouping data by grouping identical column vectors.

26. The method of claim 24, further comprising generating, for each historical user, a respective hash value by applying a hash function to the column vector representing the historical user.

27. The method of claim 26, further comprising generating the grouping data by grouping identical hash values.

28. The method of claim 19, wherein the grouping data includes a group for each unique combination of software sections executed for the historical users.

29. The method of claim 19, wherein one or more of the groups includes only a single historical user.

30. The method of claim 19, wherein the sampling data includes only a single user from one or more of the groups.

31. The method of claim 19, wherein the electronic document preparation system is a tax return preparation system.

32. The method of claim 31, wherein the historical user data includes historical user tax related data, and wherein the previously prepared electronic documents are previously prepared tax returns.

33. The method of claim 32, wherein the new calculation data includes a calculation for a tax related form associated with a tax return.

34. The method of claim 19, wherein the method further includes testing the new calculation data by executing the new calculation data for the training set data.

35. The method of claim 34, further comprising:

generating results data indicating results of testing the new calculation data; and

outputting the results data.

36. A system comprising:

at least one processor; and

at least one memory storing instructions that, when executed by the at least one processor, cause the system to;

receive, from a database, a new calculation for preparing electronic documents for users of an electronic document preparation system;

request code coverage data indicating sections of software instructions executed for a plurality of historical users of the electronic document preparation system;

generate groups of historical users based on the sections of software instructions; and

generate training set data by sampling, from each group, one or more historical users.

37. The system of claim 36, wherein trace log data identifies a data value provided or returned for a section of the software instructions executed for one or more of the historical users and wherein the groups are generated based in part on the data value.

38. The system of claim 36, wherein trace log data identifies, and the groups are generated based on, one or more of:

a sign of a data value provided or returned for a section of the software instructions executed for one or more of the historical users;

whether a result of a less than or equal to statement in the software instructions is true for being less than or true for being equal;

whether a result of a greater than or equal to statement in the software instructions is true for being greater than or true for being equal;

whether a data value was negative before being passed through an absolute value operator in the software instructions;

whether a result of a Boolean expression in the software instructions is true or false;

which variable from a group of variables passed through a minimum operator in the software instructions had the smaller data value; and

which variable from a group of variables passed through a maximum operator in the software instructions had the larger data value.

39. The system of claim 36, wherein trace log data identifies whether a result of a less than or equal to statement in the software instructions is true for being less than or true for being equal.

40. The system of claim 36, wherein trace log data identifies whether a result of a greater than or equal to statement in the software instructions is true for being greater than or true for being equal.

41. A system for generating efficient training sets for testing new processes for preparing electronic documents for users of an electronic document preparation system, the system comprising:

at least one processor; and

at least one memory coupled to the at least one processor, the at least one memory having stored therein instructions which, when executed by any set of the one or more processors, perform a process including;

receiving new calculation data related to a new calculation for generating data values for preparing electronic documents for users of an electronic document preparation system;

retrieving historical user data including electronic documents previously prepared for a plurality of historical users;

executing software instructions for preparing electronic documents with the historical user data;

generating, for each historical user, trace log data indicating sections of the software instructions that were executed for the historical users;

generating grouping data including a plurality of groups of historical users based on the sections of the software instructions executed for the historical users; and

generating training set data by sampling, from each group in the grouping data, one or more historical users.

42. The system of claim 41, wherein the software instructions include a previous calculation corresponding to the new calculation.

43. The system of claim 41, further comprising generating code coverage data based on the trace log data, the code coverage data identifying the sections of the software code that were executed for each historical user.

44. The system of claim 43, wherein the code coverage data includes a table indicating the sections of the software instructions that were executed for a plurality of historical users.

45. The system of claim 44, wherein the table includes a matrix with rows corresponding to sections of the software instructions, columns corresponding to historical users, and data values indicating whether a software section was executed for a historical user.

46. The system of claim 45, wherein the matrix represents each historical user as a column vector with data values indicating which sections of the software instructions were executed for the historical user.

47. The system of claim 46, wherein the process further includes generating the grouping data by grouping identical column vectors.

48. The system of claim 46, wherein the process further includes generating, for each historical user, a respective hash value by applying a hash function to the column vector representing the historical user.

49. The system of claim 48, wherein the process further includes generating the grouping data by grouping identical hash values.

50. The system of claim 41, wherein the grouping data includes a group for each unique combination of software sections executed for the historical users.

51. The system of claim 41, wherein one or more of the groups includes only a single historical user.

52. The system of claim 41, wherein the sampling data includes only a single user from one or more of the groups.

53. The system of claim 41, wherein the electronic document preparation system is a tax return preparation system.

54. The system of claim 53, wherein the historical user data includes historical user tax related data, and wherein the previously prepared electronic documents are previously prepared tax returns.

55. The system of claim 54, wherein the new calculation data includes a calculation for a tax related form associated with a tax return.

56. The system of claim 41, wherein the system further includes testing the new calculation data by executing the new calculation data for the training set data.

57. The system of claim 56, wherein the process further includes:

generating results data indicating results of testing the new calculation data; and

outputting the results data.

58. The system of claim 41, wherein the sampling data includes multiple historical users from one or more of the groups.