Parallel loading of markup language data files and documents into a computer database

• US 6,631,379 B2
• Filed: 01/31/2001
• Issued: 10/07/2003
• Est. Priority Date: 01/31/2001
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Cox, David A.
• Primary Examiner(s): Corrielus, Jean M.
• Application Number:
  US09/773,196
• Publication Number:
  US 20020112224A1
• Time in Patent Office: 979 Days
• Field of Search: 707/10, 707/100, 707/101, 707/103, 707/104
• US Class Current: 1/1
• CPC Class Codes: G06F 16/86 Mapping to a database
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

An XML Data Loader which employs the SAX parsing technology, provides an extendible architecture to generate SQL for each type of XML document to be loaded, and leverages multithreading and decoupling of processes to parse an XML file and to update a database. By using object-oriented programming methodologies, the system and method create new instances of the XML data loader for each XML file to be loaded into a database. Thus, many XML data files may be processed and loaded simultaneously, which minimizes system memory requirements, improves system reliability and memory management, and reduces processing time required from the start of processing an XML file to completing its loading into a database. The invention is applicable to other types of markup language documents, as well.

Claims

1. A method for loading markup language data files and documents into a computer database, said markup language data files containing data items and elements, said database having an application programming interface for modifying contents of the database through database command statements, said method comprising the steps of:

parsing a markup language data file one element at a time according to a format description, each of said elements being immediately available for further processing without waiting for subsequent elements to be parsed;

generating one or more database command statements immediately responsive to the availability of said elements such that said command generation and parsing occur substantially simultaneously;

storing each of said generated database command statements in a database command statement queue immediately upon their generation; and

executing said queued database command statements such that a database is updated substantially simultaneously with said step of parsing a data file in order to achieve loading of data from said markup language data file into a database.

2. The method as set forth in claim 1 wherein said step of parsing a markup language data file comprises parsing an extensible markup language (“

XML”

) file.

3. The method as set forth in claim 1 wherein said step of parsing a markup language data file comprises parsing an extensible markup language (“

XML”

) file using a Simple Application Programming Interface for Extensible Markup Language (“

SAX”

) method.

4. The method as set forth in claim 1 wherein said step of generating one or more database command statements comprises generating structured query language (“

SQL”

) database application programming interface command statements.

5. The method as set forth in claim 1 wherein said step of executing said queued database command statements comprises executing structured query language (“

SQL”

) database application programming interface command statements.

6. A computer readable medium having program code stored therein for loading markup language data files and documents into a computer database in a computer system, said markup language data files containing data items and elements, said database having an application programming interface for modifying contents of the database through database command statements, said program code when executed by a computer system causing the computer system to perform the steps of:

parsing a markup language data file one element at a time according to a format description, each of said elements being immediately available for further processing without waiting for subsequent elements to be parsed;

generating one or more database command statements immediately responsive to the availability of said elements such that said command generation and parsing occur substantially simultaneously;

storing each of said generated database command statements in a database command statement queue immediately upon their generation; and

executing said queued database command statements such that a database is updated substantially simultaneously with said step of parsing a data file in order to achieve loading of data from said markup language data file into a database.

7. The computer readable media as set forth in claim 6 wherein said program code for performing the step of parsing a markup language data file comprises program code for parsing an extensible markup language (“

XML”

) file.

8. The computer readable media as set forth in claim 7 wherein said program code for performing the step of parsing a markup language data file comprises program code for parsing an extensible markup language (“

XML”

) file using a Simple Application Programming Interface for Extensible Markup Language (“

SAX”

) method.

9. The computer readable media as set forth in claim 6 wherein said program code for performing the step of generating one or more database command statements comprises program code for generating structured query language (“

SQL”

) database application programming interface command statements.

10. The computer readable media as set forth in claim 6 wherein said program code for performing the step of executing said queued database command statements comprises program code for executing structured query language (“

SQL”

) database application programming interface command statements.

11. A system for loading markup language data files and documents into a computer database in a computer, said markup language data files containing data items and elements, said database having an application programming interface for modifying contents of the database through database command statements, said system comprising:

a markup language data file loader which is adapted to parse markup language data files on an element-by-element basis, each of said elements being immediately available for further processing without waiting for subsequent elements to be parsed;

said loader being adapted to select and operate an operator extension;

at least one operator extension operable by said loader, adapted to immediately evaluate each element as it is available from said loader and to generate database application programming interface command statements such that said command generation and parsing occur substantially simultaneously; and

a database updater adapted to execute said generated database application programming interface command statements in order to modify contents of a database substantially simultaneously with said loading and command generation.

12. The system as set forth in claim 11 wherein said loader is adapted to parse the extensible markup language (“

XML”

) data files and documents.

13. The system as set forth in claim 11 wherein said operator extension comprises extendable object classes.

14. The system as set forth in claim 11 wherein said operator extension is adapted to generate structured query language (“

SQL”

) database application programming interface command statements.

15. The system as set forth in claim 11 wherein said database updater adapted to update a database by executing structured query language (“

SQL”

) database application programming interface command statements.

16. The system as set forth in claim 11 further comprising a file receptor for receiving markup language files from a computer network.

17. The system as set forth in claim 16 wherein said file receptor is adapted for receiving markup language files from a computer network via a file transfer protocol (“

FTP”

).