Parallel Processing of ETL Jobs Involving Extensible Markup Language Documents

• US 20110072319A1
• Filed: 09/24/2009
• Published: 03/24/2011
• Est. Priority Date: 09/24/2009
• Status: Active Grant

• Current Assignee: GlobalFoundries, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Padmanabhan, Sriram, Mittapalli, Srinivas Kiran, Bhide, Manish A., Agarwal, Manoj K., Kotwal, Srilakshmi
• Granted Patent: US 9,064,047 B2
• Time in Patent Office: Days
• US Class Current: 714/49
• CPC Class Codes: G06F 11/3604 Software analysis for verifying properties of programs byte-code verification G06F9/44589
• CPC Class Codes: G06F 16/86 Mapping to a database

Abstract

Techniques for running an Extract Transform Load (ETL) job in parallel on one or more processors wherein the ETL job comprises use of an extensible markup language (XML) document are provided. The techniques include receiving an XML document input, identifying a node in the XML document at which partitioning of the XML document is to begin, sending partition information to each respective processor, performing a shallow parsing of the XML document in parallel on the one or more processors, wherein each processor performs shallow parsing using the identified partition node until it reaches its identified partition, using the shallow parsing to generate the partition of the input XML document, wherein each processor generates a different partition of the same XML document, and sending each partition in streaming format to an ETL job instance.

Claims

1. A method for running an Extract Transform Load (ETL) job in parallel on one or more processors wherein the ETL job comprises use of an extensible markup language (XML) document, wherein the method comprises:

receiving an XML document input;

identifying a node in the XML document at which partitioning of the XML document is to begin;

sending partition information to each respective processor;

to performing a shallow parsing of the XML document in parallel on the one or more processors, wherein each processor performs shallow parsing using the identified partition node until it reaches its identified partition;

using the shallow parsing to generate the partition of the input XML document, wherein each processor generates a different partition of the same XML document; and

sending each partition in streaming format to an ETL job instance.

2. The method of claim 1, further comprising identifying a size of each partition to be created within the XML document, wherein each partition is created on a different processor.

3. The method of claim 1, wherein the shallow parsing is performed on a single processor.

4. The method of claim 3, wherein the shallow parsing sends start and end points of each partition to the one or more processors, wherein each processor seeks to the start of its partition and sends its partition to its instance of an ETL job definition.

5. The method of claim 1, wherein the shallow parsing comprises parsing of only nodes that appear in an XML path language (XPATH) of the partition node and ignoring nodes not in the partition node XPATH.

6. The method of claim 1, further comprising ensuring that each processor performs a single pass of the XML document.

7. The method of claim 1, further comprising, if an XML schema file contains at least one of a minoccurs XML schema indicator and a maxoccurs XML schema indicator defined on a node on which partitioning is to be done:

removing the one or more indicators from the schema file;

checking validity of the indicators during the shallow parsing on a node processing a last segment of the XML document; and

generating an error if the validation fails during shallow parsing.

9. The method of claim 1, further comprising generating output in the form of a modified extract, transform, and load (ETL) job definition that can run on multiple processors in parallel.

10. The method of claim 1, wherein sending each partition in streaming format comprises adding a root node to the partition.

11. The method of claim 1, further comprising running of the ETL job comprising shredding of the XML document in parallel on multiple nodes, and wherein parallel shredding comprises using horizontal partitioning to shred different parts of the XML document in parallel on different nodes.

12. The method of claim 1, further comprising passing the XML document beyond the identified partition size as a separate XML document in streaming format.

13. The method of claim 1, further comprising partitioning the XML document for load balancing, comprising partitioning the XML document to keep an overall load on all parallel processors evenly distributed to achieve maximum performance gains.

14. The method of claim 1, further comprising distributing XML partitions evenly in a single pass manner.

15. The method of claim 1, further comprising, if a sub-tree rooted at the identified node does not contain all output nodes, adding each missing output node to the partition and modifying each XML path language (XPath) provided to the ETL job instance.

16. The method of claim 1, further comprising, if there are multiple output nodes in the XML document that are not part of a repetition path, keeping only a first occurrence of such nodes in the XML document.

17. The method of claim 1, further comprising providing a system, wherein the system comprises one or more distinct software modules, each of the one or more distinct software modules being embodied on a tangible computer-readable recordable storage medium, and wherein the one or more distinct software modules comprise a partition node identification module, a partition size computation module, a shallow parser module and an ETL job instance module executing on a hardware processor.

8. The method of 1, further comprising, if an XML schema file contains at least one of an all XML schema indicator, sequence XML schema indicator, and choice XML schema indicator defined on the node on which partitioning is to be done:

sending each node taking part in the one or more indicators but not in a repetition element path, only to a first node; and

sending each remaining node a first value of one or more output nodes that are not in the repetition path, and a schema file that does not have the XML schema indicators.

18. A computer program product comprising a tangible computer readable recordable storage medium including computer useable program code for running an Extract Transform Load (ETL) job in parallel on one or more processors wherein the ETL job comprises use of an extensible markup language (XML) document, the computer program product including:

computer useable program code for receiving an XML document input;

computer useable program code for identifying a node in the XML document at which partitioning of the XML document is to begin;

computer useable program code for sending partition information to each respective processor;

computer useable program code for performing a shallow parsing of the XML document in parallel on the one or more processors, wherein each processor performs shallow parsing using the identified partition node until it reaches its identified partition;

computer useable program code for using the shallow parsing to generate the partition of the input XML document, wherein each processor generates a different partition of the same XML document; and

computer useable program code for sending each partition in streaming format to an ETL job instance.

19. The computer program product of claim 18, further comprising computer useable program code for identifying a size of each partition to be created within the XML document, wherein each partition is created on a different processor.

20. A system for running an Extract Transform Load (ETL) job in parallel on one or more processors wherein the ETL job comprises use of an extensible markup language (XML) document, comprising:

a memory; and

at least one processor coupled to the memory and operative to;

receive an XML document input;

identify a node in the XML document at which partitioning of the XML document is to begin;

send partition information to each respective processor;

perform a shallow parsing of the XML document in parallel on the one or more processors, wherein each processor performs shallow parsing using the identified partition node until it reaches its identified partition;

use the shallow parsing to generate the partition of the input XML document, wherein each processor generates a different partition of the same XML document; and

send each partition in streaming format to an ETL job instance.