ELECTRONIC DOCUMENT SOURCE INGESTION FOR NATURAL LANGUAGE PROCESSING SYSTEMS

• US 20140164407A1
• Filed: 12/10/2012
• Published: 06/12/2014
• Est. Priority Date: 12/10/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Dubbels, Joel C.
• Granted Patent: US 9,053,085 B2
• Time in Patent Office: Days
• US Class Current: 707/755
• CPC Class Codes: G06F 40/154 Tree transformation for tree-structured or markup documents, e.g. XSLT, XSL-FO or stylesheets
• CPC Class Codes: G06F 40/205 Parsing

Abstract

The data store for a natural-language computing system may include information that originates from a plurality of different data sources—e.g., journals, websites, magazines, reference books, and the like. In one embodiment, the information or text from the data sources are converted into a single, shared format and stored as objects in a data store. In order to ingest the different documents with their respective formats, a natural language processing system may perform preprocessing to change the different formats into a normalized format. When a new text document is received, the text may be correlated to a particular properties file which includes instructions specifying how the preprocessor should interpret the received text. Based on these instructions, a preprocessor identifies relevant portions of the text document and assigns these portions to formatting elements in the normalized format. The text may then be stored in the objects based on this assignment.

Claims

1. A system, comprising:

a computer processor; and

a memory containing a program that, when executed on the computer processor, performs an operation for processing data, comprising;

receiving a plurality of electronic documents, wherein the electronic documents are arranged according to different, respective formats;

identifying a properties file associated with one of the electronic documents, the properties file defining a formatting element of the respective format in the one electronic document and an action corresponding to a text portion associated with the formatting element;

parsing the one electronic document to identify the formatting element;

upon identifying the text portion associated with the formatting element, performing the action to the text portion by assigning the text portion to a formatting element of a normalized format; and

storing the text portion into a NLP object based on the formatting element of the normalized format, wherein text in the NLP object is arranged based on the normalized format.

2. The system of claim 1, wherein the properties file is one of a plurality of properties files, wherein each properties file is associated with one of the respective formats of the electronic documents.

3. The system of claim 1, further comprising:

instantiating a preprocessor for parsing the one electronic document based on a extension class defined in the properties file, wherein the extension class is configured to parse only electronic documents arranged using the respective format.

4. The system of claim 1, wherein the formatting element of the respective format is one of:

a header, a title, and a section in the one electronic document.

5. The system of claim 1, wherein the NLP object comprises text portions retrieved from other ones of the plurality of electronic documents, wherein the text portions are assigned to the formatting element of the normalized format.

6. The system of claim 5, wherein the NLP object is a common analysis system (CAS) data structure.

7. The system of claim 1, further comprising:

annotating the text in the NLP object for use in a natural-language computing system where the natural-language computing system uses the annotated text to communicate with a user.

13. The computer program product of claim 1, further comprising computer-readable program code configured to:

annotate the text in the NLP object for use in a natural-language computing system where the natural-language computing system uses the annotated text to communicate with a user.

8. A computer program product comprising:

a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code comprising computer-readable program code configured to;

receive a plurality of electronic documents, wherein the electronic documents are arranged according to different, respective formats;

identify a properties file associated with one of the electronic documents, the properties file defining a formatting element of the respective format in the one electronic document and an action corresponding to a text portion associated with the formatting element;

parse the one electronic document to identify the formatting element;

upon identifying the text portion associated with the formatting element, perform the action to the text portion by assigning the text portion to an formatting element of a normalized format; and

store the text portion into a NLP object based on the formatting element of the normalized format, wherein text in the NLP object is arranged based on the normalized format.

9. The computer program product of claim 8, wherein the properties file is one of a plurality of properties files, wherein each properties file is associated with one of the respective formats of the electronic documents.

10. The computer program product of claim 8, further comprising computer-readable program code configured to:

instantiate a preprocessor for parsing the one electronic document based on a extension class defined in the properties file, wherein the extension class is configured to parse only electronic documents arranged using the respective format.

11. The computer program product of claim 8, wherein the formatting element of the respective format is one of:

a header, a title, and a section in the one electronic document.

12. The computer program product of claim 8, wherein the NLP object comprises text portions retrieved from other ones of the plurality of electronic documents, wherein the text portions are assigned to the formatting element of the normalized format, and wherein the NLP object is a common analysis system (CAS) data structure.