Table item information extraction with continuous machine learning through local and global models

• US 10,241,992 B1
• Filed: 04/27/2018
• Issued: 03/26/2019
• Est. Priority Date: 04/27/2018
• Status: Active Grant

• Current Assignee: Open Text Sa Ulc (Open Text Corporation)
• Original Assignee: Open Text Sa Ulc (Open Text Corporation)
• Inventors: Middendorf, Mattias Theodor, Hammann, Gisela Barbara Cacilie, Peust, Carsten
• Primary Examiner(s): McIntosh, Andrew T
• Application Number:
  US15/964,654
• Time in Patent Office: 333 Days
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G06F 40/177 of tables; using ruled lines
• CPC Class Codes: G06F 40/186 Templates
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: G06V 30/412 Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors

Abstract

A bipartite application implements a table auto-completion (TAC) algorithm on the client side and the server side. A client module runs a local model of the TAC algorithm on a user device and a server module runs a global model of the TAC algorithm on a server machine. The local model is continuously adapted through on-the-fly training, with as few as a negative example, to perform TAC on the client side, one document at a time. Knowledge thus learned by the local model is used to improve the global model on the server side. The global model can be utilized to automatically and intelligently extract table information from a large number of documents with significantly improved accuracy, requiring minimal human intervention even on complex tables.

Claims

1. A method, comprising:

displaying, on a user device through a user interface of a bipartite application, a database table and an image, the database table having a plurality of columns, the image containing a table, the bipartite application implemented on the user device and a server machine in a backend of an enterprise computing environment, the displaying performed by a client module of the bipartite application executing on the user device, the client module including a local model of a table auto-completion algorithm, the user interface including a user interface element associated with the table auto-completion algorithm;

responsive to a user selecting the user interface element displayed on the user device, performing, by the client module running the local model of the table auto-completion algorithm;

analyzing a portion of the table highlighted by the user on the user interface, the portion of the table highlighted by the user on the user interface defining initial coordinates on the user interface;

determining a data point for each column of the database table using the initial coordinates;

automatically extracting data points thus determined from the table utilizing the local model;

entering the data points automatically extracted from the table utilizing the local model into the plurality of columns of the database table; and

storing information about the data points in the local model as positive examples;

determining, by the client module running the local model of the table auto-completion algorithm utilizing the positive examples in the local model, a plurality of additional data points in the table;

automatically extracting the plurality of additional data points from the table utilizing the local model and entering the plurality of additional data points extracted from the table utilizing the local model into the plurality of columns of the database table;

receiving, by the client module running the local model of the table auto-completion algorithm, a correction to a data point of the plurality of additional data points automatically extracted from the table utilizing the local model;

correcting the local model utilizing the correction to the data point and including the data point in the local model as a negative example, the correcting performed by the client module running the local model of the table auto-completion algorithm;

automatically continuously extracting table information from the table utilizing the positive and negative examples in the local model until extraction of the table information from the table is completed;

communicating the local model from the client module to a server module of the bipartite application running on the server machine in the backend of the enterprise computing environment, the server module including a global model of a table auto-completion algorithm;

updating the global model of the table auto-completion algorithm utilizing the local model;

automatically extracting table information from a plurality of documents, the automatically extracting performed by the server module executing on the server machine utilizing the global model; and

automatically entering into database fields the table information extracted from the plurality of documents utilizing the global model, the automatically entering performed by the server module executing on the server machine.

2. The method according to claim 1, wherein the table comprises a complex table having a plurality of items, each item of the plurality of items containing multiple lines or rows.

3. The method according to claim 1, further comprising:

detecting a document type of the image.

4. The method according to claim 3, wherein the plurality of documents is classified as having the document type.

5. The method according to claim 1, wherein the local model comprises a cell model defining properties of cells of a given column, a line model defining a number of cells per line and transition there-between, and a document model defining line distances and a number of lines per document and wherein the plurality of additional data points is automatically extracted from the table utilizing the cell model, the line model, and the document model.

6. The method according to claim 1, wherein the global model comprises a cell model defining properties of cells of a given column, a line model defining a number of cells per line and transition there-between, and a document model defining line distances and a number of lines per document and wherein the table information is automatically extracted from the plurality of documents utilizing the cell model, the line model, and the document model.

7. The method according to claim 1, wherein the local model comprises a plurality of features describing a layout of a document, the layout including an orientation of the document.

8. A system, comprising:

a processor;

a non-transitory computer-readable medium; and

stored instructions translatable by the processor to perform;

displaying, on a user device, a database table and an image, the database table having a plurality of columns, the image containing a table, the user interface including a user interface element for table auto-completion;

responsive to a user selecting the user interface element displayed on the user device, performing;

analyzing a portion of the table highlighted by the user on the user interface, the portion of the table highlighted by the user on the user interface defining initial coordinates on the user interface;

determining a data point for each column of the database table using the initial coordinates;

automatically extracting data points thus determined from the table utilizing the local model;

entering the data points automatically extracted from the table utilizing the local model into the plurality of columns of the database table; and

storing information about the data points in a local model of a table auto-completion algorithm as positive examples;

determining a plurality of additional data points in the table utilizing the positive examples in the local model;

automatically extracting the plurality of additional data points from the table utilizing the local model and entering the plurality of additional data points extracted from the table utilizing the local model into the plurality of columns of the database table;

receiving a correction to a data point of the plurality of additional data points automatically extracted from the table utilizing the local model;

correcting the local model utilizing the correction to the data point and including the data point in the local model as a negative example;

automatically continuously extracting table information from the table utilizing the positive and negative examples in the local model until extraction of the table information from the table is completed;

updating a global model of the table auto-completion algorithm utilizing the local model;

automatically extracting table information from a plurality of documents utilizing the global model; and

automatically entering into database fields the table information extracted from the plurality of documents utilizing the global model.

9. The system of claim 8, wherein the table comprises a complex table having a plurality of items, each item of the plurality of items containing multiple lines or rows.

10. The system of claim 8, wherein the image and the plurality of documents are associated with a document type.

11. The system of claim 8, wherein the local model comprises a cell model defining properties of cells of a given column, a line model defining a number of cells per line and transition there-between, and a document model defining line distances and a number of lines per document and wherein the plurality of additional data points is automatically extracted from the table utilizing the cell model, the line model, and the document model.

12. The system of claim 8, wherein the global model comprises a cell model defining properties of cells of a given column, a line model defining a number of cells per line and transition there-between, and a document model defining line distances and a number of lines per document and wherein the table information is automatically extracted from the plurality of documents utilizing the cell model, the line model, and the document model.

13. The system of claim 8, wherein the local model comprises a plurality of features describing a layout of a document, the layout including an orientation of the document.

14. A computer program product comprising a non-transitory computer-readable medium storing instructions translatable by a processor to perform:

displaying, on a user device, a database table and an image, the database table having a plurality of columns, the image containing a table, the user interface including a user interface element for table auto-completion;

responsive to a user selecting the user interface element displayed on the user device, performing;

analyzing a portion of the table highlighted by the user on the user interface, the portion of the table highlighted by the user on the user interface defining initial coordinates on the user interface;

determining a data point for each column of the database table using the initial coordinates;

automatically extracting data points thus determined from the table utilizing the local model;

entering the data points automatically extracted from the table utilizing the local model into the plurality of columns of the database table; and

storing information about the data points in a local model of a table auto-completion algorithm as positive examples;

determining a plurality of additional data points in the table utilizing the positive examples in the local model;

automatically extracting the plurality of additional data points from the table utilizing the local model and entering the plurality of additional data points extracted from the table utilizing the local model into the plurality of columns of the database table;

receiving a correction to a data point of the plurality of additional data points automatically extracted from the table utilizing the local model;

correcting the local model utilizing the correction to the data point and including the data point in the local model as a negative example;

automatically continuously extracting table information from the table utilizing the positive and negative examples in the local model until extraction of the table information from the table is completed;

updating a global model of the table auto-completion algorithm utilizing the local model;

automatically extracting table information from a plurality of documents utilizing the global model; and

automatically entering into database fields the table information extracted from the plurality of documents utilizing the global model.

15. The computer program product of claim 14, wherein the table comprises a complex table having a plurality of items, each item of the plurality of items containing multiple lines or rows.

16. The computer program product of claim 14, wherein the instructions are further translatable by the processor to perform:

detecting a document type of the image.

17. The computer program product of claim 16, wherein the plurality of documents is classified as having the document type.

18. The computer program product of claim 14, wherein the local model comprises a cell model defining properties of cells of a given column, a line model defining a number of cells per line and transition there-between, and a document model defining line distances and a number of lines per document and wherein the plurality of additional data points is automatically extracted from the table utilizing the cell model, the line model, and the document model.

19. The computer program product of claim 14, wherein the global model comprises a cell model defining properties of cells of a given column, a line model defining a number of cells per line and transition there-between, and a document model defining line distances and a number of lines per document and wherein the table information is automatically extracted from the plurality of documents utilizing the cell model, the line model, and the document model.

20. The computer program product of claim 14, wherein the local model comprises a plurality of features describing a layout of a document, the layout including an orientation of the document.