NATURAL LANGUAGE PROCESSING-ASSISTED EXTRACT, TRANSFORM, AND LOAD TECHNIQUES

• US 20160117293A1
• Filed: 04/21/2015
• Published: 04/28/2016
• Est. Priority Date: 10/23/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: DETTMAN, Elizabeth T., DUBBELS, Joel C., FREED, Andrew R., PAYNE, Michael T., SCHROEDER, Michael W.
• Granted Patent: US 10,127,201 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]
• CPC Class Codes: G06F 40/151 Transformation
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

Embodiments presented herein disclose techniques for transforming input documents having disparate formats into a normalized format (e.g., Atom, RSS, HTML, customized XML, etc.). According to one embodiment, a plurality of fields is identified in an input document that has a given format. Each field includes a descriptor and text content associated with the descriptor. For each field, semantic properties are evaluated for the descriptor and text content against a plurality of mapping rules to determine whether the field is consistent with one of a plurality of fields of a target format. Each mapping rule specifies characteristics associated with one of the fields in the target format. Once so determined, a mapping from the first field to the second field is defined.

Claims

1. A method for mapping fields of an input document structured according to a first format, the method comprising:

identifying a plurality of first fields in the input document, wherein each first field includes a descriptor and text content associated with the descriptor; and

for each first field;

evaluating, via one or more natural language processing techniques, semantic properties of the descriptor and the text content against a plurality of mapping rules to determine whether the first field is consistent with one of a plurality of second fields in a target format, wherein each mapping rule specifies characteristics associated with one of the second fields in the target format, and

upon determining that the first field is consistent with one of the second fields, defining a mapping from the first field to the second field determined to be consistent with the first field.

2. The method of claim 1, further comprising:

mapping the text content of each first field determined to be consistent with a corresponding second field to a target document formatted based on the target format; and

sending the target document to a target system.

3. The method of claim 1, further comprising, converting the first format of the input document into the target format based on the defined mappings.

4. The method of claim 3, further comprising, sending the input document having the target format to a target system.

5. The method of claim 1, wherein the target format is an Atom syndication format.

6. The method of claim 1, wherein evaluating semantic properties of the descriptor and the text content against the plurality of mapping rules to determine whether the first field is consistent with one of the plurality of second fields in the target format comprises:

determining, for each mapping rule, a descriptor score associated with the descriptor and a content score associated with the text content, the descriptor score and the content score indicating a likelihood that the respective descriptor and text content match the characteristics specified in the mapping rule; and

converging the descriptor score and the content score into a consolidated score based on a weighting between the descriptor score and the content score specified by the associated mapping rule.

7. The method of claim 6, wherein determining that the first field is consistent with one of the second fields comprises:

determining whether the consolidated score for the mapping rule associated with one of the second fields exceeds a specified threshold.