System and method for selecting data sample groups for machine learning of context of data fields for various document types and/or for test data generation for quality assurance systems

• US 10,140,277 B2
• Filed: 10/13/2016
• Issued: 11/27/2018
• Est. Priority Date: 07/15/2016
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Unsal, Cem
• Primary Examiner(s): Tapp, Amelia
• Application Number:
  US15/292,510
• Publication Number:
  US 20180018310A1
• Time in Patent Office: 775 Days
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/013 Automatic theorem proving
• CPC Class Codes: G06N 5/041 Abduction
• CPC Class Codes: G06Q 40/123 Tax preparation or submission

Abstract

A method and system learns new forms to be incorporated into an electronic document preparation system. The method and system receive form data related to a new form having a plurality of data fields that expect data values based on specific functions. The method and system gather training set data including previously filled forms having completed data fields corresponding to the data fields of the new form. The method and system group the training set data into groups and sample the groups. The method and system utilize machine learning in conjunction with the sampled training set data to identify an acceptable function for each of the data fields of the new form. The grouped and sampled training set data can also be passed to a quality assurance system.

Claims

1. A computing system implemented method for efficiently learning new forms in an electronic document preparation system, the method comprising:

receiving form data related to a new form having a plurality of data fields;

gathering training set data related to previously filled forms, each previously filled form having one or more completed data fields that correspond to a respective data field of the new form;

deleting from the training set data one or more sets of data of a previously filled form where a first set of data of the previously filled form matched a second set of data of the previously filled form and the deleted training set data includes the second set of data;

generating, for a first selected data field, dependency data indicating one or more possible dependencies for an acceptable function, the possible dependencies including one or more data fields of the new form other than the first selected data field, the possible dependencies further including one or more constants of the first selected data field, the possible dependencies further including one or more values of data fields from a form other than the new form;

generating, for a first selected data field of the plurality of data fields of the new form and based on the dependency data, candidate function data including a plurality of candidate functions;

generating, for the first selected data field and based on the dependency data, grouping data by forming a plurality of groups from the training set data based on respective categories and assigning each of a plurality of the previously filled forms to a respective one of the groups based on the categories;

generating, for the first selected data field, sampling data by selecting one or more previously filled forms from each group;

generating, for each candidate function, test data by applying the candidate function to a portion of the training set data corresponding to the sampling data related to the candidate function;

identifying one or more candidate functions of the plurality of candidate functions that have associated test data that are a best match to the training set data as compared with other candidate functions of the plurality of candidate functions;

generating one or more additional candidate functions, the additional candidate functions being based on the identified one or more candidate functions that have associated test data that are a best match;

repeatedly identifying generated candidate functions that have associated test data that are a best match to the training set data and generating one or more additional candidate functions, the additional candidate functions being based on the identified one or more candidate functions that have associated test data that are a best match until one or more candidate functions are determined to have associated test data that matches the training set data with a predetermined tolerance;

identifying, from the plurality of candidate functions, an acceptable function for the first selected data field by comparing the test data to the training set data and identifying test data that matches the training set data within a predetermined tolerance, the identified acceptable function being a candidate function associated with the matching test data; and

generating and outputting results data indicating the acceptable function for the first data field of the new form.

2. The method of claim 1, wherein the possible dependencies include one or more of:

a data field from the new form;

a data field from one or more forms other than the new form; and

a constant.

3. The method of claim 2, wherein generating grouping data includes:

identifying previously filled forms having identical combinations of data values in data fields related to the dependency data; and

excluding from the groups previously filled forms having identical combinations of data values in data fields related to the dependency data.

4. The method of claim 3, wherein the data fields related to the dependency data include data fields of the previously filled forms that correspond to the first selected data field.

5. The method of claim 3 wherein the data fields related to the dependency data include data fields from the previously filled forms or from other forms or worksheets related to the previously filled forms.

6. The method of claim 5, wherein the groups are based on respective signs of data values in data fields related to the dependency data.

7. The method of claim 5, wherein the groups are based on magnitudes of data values in data fields related to the dependency data.

8. The method of claim 5, wherein the groups are based on relationships of one or more constants to data values in data fields related to the dependency data.

9. The method of claim 5, wherein the groups are based on one or of magnitudes of data values in data fields related to the dependency data, signs of data values in data fields related to the dependency data, and a relationship of one or more constants to data values in data fields related to the dependency data.

10. The method of claim 1, wherein the groups are selected to ensure that the sampling data will include previously filled forms having extreme data values in data fields corresponding to the selected data field or in data fields included in one or more of the candidate functions.

11. The method of claim 1, further comprising, after identifying the acceptable function for the first selected data field of the new form, identifying a second acceptable function for a second selected data field from the plurality of data fields of the new form.

12. The method of claim 11, further comprising:

generating, for the second selected data field, grouping data by forming a plurality of groups from the training set data based on respective categories and assigning each of a plurality of the previously filled forms to one of the groups based on the categories;

generating second sampling data by selecting one or more previously filled forms from each group;

generating, for the second selected data field, second candidate function data including a plurality of second candidate functions;

generating, for each second candidate function, second test data by applying the second candidate function to a portion of the training set data corresponding to the second sampling data;

identifying, from the plurality of functions, the second correct candidate function for the second selected data field, by comparing the second test data to the training set data and identifying second test data that matches the training set data within a selected tolerance; and

generating and outputting second results data indicating the second acceptable function for the second selected data field of the new form.

13. The method of claim 1, wherein the new form is a finance related form and the training set data includes historical financial data related to previously prepared financial documents, the historical financial data including the previously filled forms.

14. The method of claim 13, wherein the historical financial data includes previously prepared financial documents that were previously filed with a government or financial institution.

15. The method of claim 1, wherein the training set data includes fabricated data related to fabricated documents, the fabricated data including the previously filled forms.

16. The method of claim 15, further comprising receiving the fabricated financial data from one or more third parties.

17. The method of claim 1, wherein the candidate functions each include one or more operators from a library of operators including:

an addition operator;

a subtraction operator;

a division operator;

a multiplication operator;

an exponential operator;

logical operators;

a string comparison operator; and

existence condition operators.

18. The method of claim 1, wherein the new form is a new tax form and the training set data includes previously prepared tax returns.

19. A system for efficiently learning new forms in an electronic document preparation system, the system comprising:

at least one processor; and

at least one memory coupled to the at least one processor, the at least one memory having stored therein instructions which, when executed by any set of the at least one processors, perform a process including;

receiving, with an interface module of a computing system, form data related to a new form having a plurality of data fields;

gathering, with a data acquisition module of a computing system, training set data related to previously filled forms, each previously filled form having completed data fields that each correspond to a respective data field of the new form;

deleting from the training set data one or more sets of data of a previously filled form where a first set of data of the previously filled form matched a second set of data of the previously filled form and the deleted training set data includes the second set of data;

generating, for a first selected data field, dependency data indicating one or more possible dependencies for an acceptable function, the possible dependencies including one or more data fields of the new form other than the first selected data field, the possible dependencies further including one or more constants of the first selected data field, the possible dependencies further including one or more values of data fields from a form other than the new form;

generating, with a grouping module of a computing system and for a first selected data field of the new form and based on the dependency data, grouping data by forming a plurality of groups from the training set data based on respective categories and assigning each of a plurality of the previously filled forms to a respective one of the groups;

generating, with a sampling module of a computing system, sampling data by selecting one or more previously filled forms from each group;

generating, with a machine learning module of a computing system, for the first selected data field and based on the dependency data, candidate function data including a plurality of candidate functions;

generating, with the machine learning module and for each candidate function, test data by applying the candidate function to a portion of the training set data corresponding to the sampling data;

identifying one or more candidate functions of the plurality of candidate functions that have associated test data that are a best match to the training set data as compared with other candidate functions of the plurality of candidate functions;

generating one or more additional candidate functions, the additional candidate functions being based on the identified one or more candidate functions that have associated test data that are a best match;

repeatedly identifying generated candidate functions that have associated test data that are a best match to the training set data and generating one or more additional candidate functions, the additional candidate functions being based on the identified one or more candidate functions that have associated test data that are a best match until one or more candidate functions are determined to have associated test data that matches the training set data with a predetermined tolerance;

identifying, with the machine learning module and from the plurality of candidate functions, an acceptable candidate for the first selected data field, by comparing the test data to the training set data and identifying test data that matches the training set data within a predetermined tolerance, the identified acceptable function being a candidate function associated with the matching test data;

generating, with the machine learning module, results data indicating the acceptable function for the first data field of the new form; and

outputting, with the interface module, the results data.

20. The system of claim 19, wherein generating grouping data includes:

identifying previously filled forms that are identical to each other in selected areas; and

discarding from consideration the previously filled forms that are identical in the selected areas.

21. The system of claim 20, wherein the selected areas are data fields of the previously filled forms that are related to the first selected data field.

22. The system of claim 21, wherein the process further includes generating, for a first selected data field of the plurality of data fields of the new form, dependency data indicating one or more possible dependencies for the acceptable function.

23. The system of claim 22, wherein the possible dependencies include one or more of:

a data field from the new form;

multiple data fields from the new form;

a data field from a form other than a new form;

multiple data fields from multiple forms other than the new form; and

a constant.

24. The system of claim 22, wherein the dependency data indicates multiple data fields from the previously filled forms or from other forms or worksheets related to the previously filled forms.

25. The system of claim 24, wherein the groups are based on respective signs of data values in the multiple data fields.

26. The system of claim 24, wherein the groups are based on magnitudes of data values in the multiple data fields.

27. The system of claim 24, wherein the groups are based on both magnitudes and signs of data values in the multiple data fields.

28. The system of claim 19, wherein generating the sampling data includes selecting from each group a selected number of previously filled forms.

29. The system of claim 19, wherein the groups are selected to ensure that the sampling data will include previously filled forms having extreme data values in data fields corresponding to the selected data field or in data fields included in one or more of the candidate functions.

30. The system of claim 19, wherein the process further includes, after identifying the acceptable function for the first selected data field of the new form, identifying a second acceptable function for a second selected data field from the plurality of data fields of the new form.

31. The system of claim 30, wherein the process further includes:

generating, for the second selected data field, grouping data by forming a plurality of groups from the training set data based on respective categories and assigning each of a plurality of the previously filled forms to one of the groups;

generating second sampling data by selecting one or more previously filled forms from each group;

generating, for the second selected data field, second candidate function data including a plurality of second candidate functions;

generating, for each second candidate function, second test data by applying the second candidate function to a portion of the training set data corresponding to the second sampling data;

identifying, from the plurality of functions, the second correct candidate function for the second selected data field, by comparing the second test data to the training set data and identifying second test data that matches the training set data within a selected tolerance;

generating second results data indicating the second acceptable function for the second selected data field of the new form; and

outputting the second results data.