Building classification and extraction models based on electronic forms

• US 10,140,511 B2
• Filed: 12/30/2016
• Issued: 11/27/2018
• Est. Priority Date: 03/13/2013
• Status: Active Grant

• Current Assignee: Kofax Incorporated
• Original Assignee: Kofax Incorporated
• Inventors: Macciola, Anthony, Amtrup, Jan W., Thompson, Stephen Michael
• Primary Examiner(s): Shah, Utpal
• Application Number:
  US15/396,322
• Publication Number:
  US 20170109610A1
• Time in Patent Office: 697 Days
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/1452 based on positionally close symbols, e.g. amount sign or URL-specific characters
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors

Abstract

According to one embodiment, a computer-implemented method is configured for building a classification and/or data extraction knowledge base using an electronic form. The method includes: receiving an electronic form having associated therewith a plurality of metadata labels, each metadata label corresponding to at least one element of interest represented within the electronic form; parsing the plurality of metadata labels to determine characteristic features of the element(s) of interest; building a representation of the electronic form based on the plurality of metadata labels; generating a plurality of permutations of the representation of the electronic form by applying a predetermined set of variations to the representation; and training either a classification model, an extraction model, or both using: the representation of the electronic form, and the plurality of permutations of the representation of the electronic form. Corresponding systems and computer program products are also disclosed.

Claims

1. A computer-implemented method for building a classification and/or data extraction knowledge base using an electronic form, the method comprising:

receiving an electronic form having associated therewith a plurality of metadata labels, each metadata label corresponding to at least one element of interest represented within the electronic form;

parsing the plurality of metadata labels to determine characteristic features of the element(s) of interest;

building a representation of the electronic form based on the plurality of metadata labels;

generating a plurality of permutations of the representation of the electronic form by applying a predetermined set of variations to the representation; and

training either a classification model, an extraction model, or both using;

the representation of the electronic form, and

the plurality of permutations of the representation of the electronic form.

2. The computer-implemented method as recited in claim 1, comprising:

classifying, using the classification model, one or more images each independently depicting a document as belonging to a same class of document as the electronic form.

3. The computer-implemented method as recited in claim 1, comprising:

extracting, using the extraction model, information from one or more images each independently depicting a document belonging to a same class of document as the electronic form.

4. The computer-implemented method as recited in claim 3, comprising:

generating a second electronic form comprising a plurality of elements of interest, each element of interest in the second electronic form corresponding to at least one of the elements of interest in the electronic form; and

associating the information extracted from the one or more images with appropriate ones of the plurality of elements of interest in the second electronic form.

5. The computer-implemented method as recited in claim 1, wherein the electronic form comprises machine-printed text, handwritten text, or both represented in association with at least one of the elements of interest;

and the method comprising;

determining a data type associated with one or more of the elements of interest based on analyzing the machine-printed text associated with the one or more of the elements of interest, the handwritten text associated with the one or more of the elements of interest, or both.

6. The method as recited in claim 1, comprising determining a data type associated with one or more of the elements of interest based on parsing the plurality of metadata labels.

7. The computer-implemented method as recited in claim 1, wherein the representation of the electronic form comprises a definition of one or more characteristic features of the electronic form.

8. The computer-implemented method as recited in claim 7, wherein the characteristic features are selected from:

invariant information represented on the electronic form and associated text patterns;

field locations for each field represented on the electronic form;

field names for each field represented on the electronic form;

data types for each field represented on the electronic form; and

image features descriptive of corresponding element(s) of interest represented on the electronic form.

9. The computer-implemented method as recited in claim 1, wherein the representation comprises an XML document defining a plurality of elements and a plurality of fields of the electronic form.

10. The computer-implemented method as recited in claim 1, wherein the plurality of permutations are further based on at least one of:

the electronic form, some or all of the elements of interest represented within the electronic form, and the parsed metadata labels.

11. A system comprising a processor configured to execute logic, the logic being configured, upon execution thereof by the processor, to cause the processor to perform a computer-implemented method comprising:

receiving an electronic form having associated therewith a plurality of metadata labels, each metadata label corresponding to at least one element of interest represented within the electronic form;

parsing the plurality of metadata labels to determine characteristic features of the element(s) of interest;

building a representation of the electronic form based on the plurality of metadata labels;

generating a plurality of permutations of the representation of the electronic form by applying a predetermined set of variations to the representation; and

training either a classification model, an extraction model, or both using;

the representation of the electronic form, and

the plurality of permutations of the representation of the electronic form.

12. A computer program product comprising a computer readable storage medium having embodied thereon computer readable program instructions configured to cause a mobile device, upon execution of the computer readable program instructions, to perform operations comprising:

receiving an electronic form having associated therewith a plurality of metadata labels, each metadata label corresponding to at least one element of interest represented within the electronic form;

parsing the plurality of metadata labels to determine characteristic features of the element(s) of interest;

building a representation of the electronic form based on the plurality of metadata labels;

generating a plurality of permutations of the representation of the electronic form by applying a predetermined set of variations to the representation; and

training either a classification model, an extraction model, or both using;

the representation of the electronic form, and

the plurality of permutations of the representation of the electronic form.

13. The computer program product as recited in claim 12, comprising computer readable program instructions configured to cause the mobile device, upon execution of the computer readable program instructions, to:

classify, using the classification model, one or more images each independently depicting a document as belonging to a same class of document as the electronic form.

14. The computer program product as recited in claim 12, comprising computer readable program instructions configured to cause the mobile device, upon execution of the computer readable program instructions, to:

extract, using the extraction model, information from one or more images each independently depicting a document belonging to a same class of document as the electronic form.

15. The computer program product as recited in claim 14, comprising computer readable program instructions configured to cause the mobile device, upon execution of the computer readable program instructions, to:

generate a second electronic form comprising a plurality of elements of interest, each element of interest in the second electronic form corresponding to at least one of the elements of interest in the electronic form; and

associate the information extracted from the one or more images with appropriate ones of the plurality of elements of interest in the second electronic form.

16. The computer program product as recited in claim 12, wherein the electronic form comprises machine-printed text, handwritten text, or both represented in association with at least one of the elements of interest;

and the method comprising;

determining a data type associated with one or more of the elements of interest based on analyzing the machine-printed text associated with the one or more of the elements of interest, the handwritten text associated with the one or more of the elements of interest, or both.

17. The computer program product as recited in claim 12, comprising computer readable program instructions configured to cause the mobile device, upon execution of the computer readable program instructions, to:

determine a data type associated with one or more of the elements of interest based on parsing the plurality of metadata labels.

18. The computer program product as recited in claim 12, wherein the representation of the electronic form comprises a definition of one or more characteristic features of the electronic form.

19. The computer program product as recited in claim 12, wherein the representation comprises an XML document defining a plurality of elements and a plurality of fields of the electronic form.

20. The computer program product as recited in claim 12, wherein the plurality of permutations are further based on at least one of:

the electronic form, some or all of the elements of interest represented within the electronic form, and the parsed metadata labels.