Electronic document source ingestion for natural language processing systems
First Claim
1. A system, comprising:
- a computer processor; and
a memory containing a program that, when executed on the computer processor, performs an operation for processing data, comprising;
receiving a plurality of electronic documents, wherein each electronic document is arranged according to a different, respective format comprising a plurality of headers;
identifying a properties file associated with one of the electronic documents, the properties file defining a particular header of the respective format in the one electronic document, an action corresponding to a text portion associated with the particular header, and an extension class;
instantiating a preprocessor for parsing the one electronic document based on the extension class, wherein the preprocessor is configured to parse only electronic documents arranged using the respective format;
parsing the one electronic document to identify the particular header using the preprocessor;
upon identifying the text portion associated with the particular header, performing the action to the text portion by assigning the text portion to a formatting element of a normalized format; and
storing the text portion into a natural language processing (NLP) object based on the formatting element of the normalized format, wherein text in the NLP object is arranged based on the normalized format.
1 Assignment
0 Petitions
Accused Products
Abstract
The data store for a natural-language computing system may include information that originates from a plurality of different data sources—e.g., journals, websites, magazines, reference books, and the like. In one embodiment, the information or text from the data sources are converted into a single, shared format and stored as objects in a data store. In order to ingest the different documents with their respective formats, a natural language processing system may perform preprocessing to change the different formats into a normalized format. When a new text document is received, the text may be correlated to a particular properties file which includes instructions specifying how the preprocessor should interpret the received text. Based on these instructions, a preprocessor identifies relevant portions of the text document and assigns these portions to formatting elements in the normalized format. The text may then be stored in the objects based on this assignment.
-
Citations
13 Claims
-
1. A system, comprising:
-
a computer processor; and a memory containing a program that, when executed on the computer processor, performs an operation for processing data, comprising; receiving a plurality of electronic documents, wherein each electronic document is arranged according to a different, respective format comprising a plurality of headers; identifying a properties file associated with one of the electronic documents, the properties file defining a particular header of the respective format in the one electronic document, an action corresponding to a text portion associated with the particular header, and an extension class; instantiating a preprocessor for parsing the one electronic document based on the extension class, wherein the preprocessor is configured to parse only electronic documents arranged using the respective format; parsing the one electronic document to identify the particular header using the preprocessor; upon identifying the text portion associated with the particular header, performing the action to the text portion by assigning the text portion to a formatting element of a normalized format; and storing the text portion into a natural language processing (NLP) object based on the formatting element of the normalized format, wherein text in the NLP object is arranged based on the normalized format. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A computer program product comprising:
a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising computer-readable program code configured to; receive a plurality of electronic documents, wherein each electronic document is arranged according to a different, respective format comprising a plurality of headers; identify a properties file associated with one of the electronic documents, the properties file defining a particular header of the respective format in the one electronic document, an action corresponding to a text portion associated with the particular header, and an extension class; instantiate a preprocessor for parsing the one electronic document based on the extension class, wherein the preprocessor is configured to parse only electronic documents arranged using the respective format; parse the one electronic document to identify the particular header using the preprocessor; upon identifying the text portion associated with the particular header, perform the action to the text portion by assigning the text portion to a formatting element of a normalized format; and store the text portion into a natural language processing (NLP) object based on the formatting element of the normalized format, wherein text in the NLP object is arranged based on the normalized format. - View Dependent Claims (9, 10, 11, 12, 13)
Specification