BUSINESS INFORMATION WAREHOUSE TOOLKIT AND LANGUAGE FOR WAREHOUSING SIMPLIFICATION AND AUTOMATION

• US 20080307386A1
• Filed: 06/07/2007
• Published: 12/11/2008
• Est. Priority Date: 06/07/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Wang, Rui, He, Bin, CHEN, YING, Rhodes, James J., Lelescu, Ana
• Granted Patent: US 8,056,054 B2
• Time in Patent Office: Days
• US Class Current: 717/109
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising

Abstract

A method for use with an information (or data) warehouse comprises managing the information warehouse with instructions in a declarative language. The instructions specify information warehouse-level tasks to be done without specifying certain details of how the tasks are to be implemented, for example, using databases and text indexers. The details are hidden from the user and include, for example, in an information warehouse having a FACT table that joins two or more dimension tables, details of database level operations when structured data are being handled, including database command line utilities, database drivers, and structured query language (SQL) statements; and details of text-indexing engines when unstructured data are being handled. The information warehouse is managed in a dynamic way in which different tasks—such as data loading tasks and information warehouse construction tasks—may be interleaved (i.e., there is no particular order in which the different tasks must be completed).

Claims

1. A method for information warehouse automation, comprising:

providing a language specification for information warehouse construction, data loading in an information warehouse (IW), and data maintenance of the IW, the language specification having IW level commands, dimension level commands, source mapping commands, data loading commands, failure recovery commands, and error correction commands;

constructing the IW based on a script comprising at least one of the IW level commands and the dimension level commands according to the language specification for information warehouse construction;

loading data into the IW based on the script comprising at least one of the source mapping commands and data loading commands according to the language specification for data loading;

performing an IW maintenance operation on the IW based on the script comprising at least one of the failure recovery commands and error correction commands according to the language specification for IW maintenance;

applying the IW level commands defining a language syntax at the granularity of the information warehouse, wherein system and database level tasks required to perform the commands are transparent to a user of the IW level commands;

applying the dimension level commands defining a language syntax at the granularity of facts and dimensions, wherein system and database level tasks required to perform the commands are transparent to a user of the dimension level commands;

implementing IW level commands including commands for creating the IW, using the IW, and dropping the IW;

implementing dimension level commands including commands for creating a dimension, altering the dimension, and dropping the dimension;

applying the IW level commands and the dimension level commands including implementing support for both structured and unstructured data;

implementing source mapping commands and data loading commands defining a language syntax wherein system and database level tasks required to perform the commands are transparent to a user of the source mapping commands and data loading commands;

implementing the source mapping commands and data loading commands including performing a data extraction using an xpath to locate a desired data field;

implementing the source mapping commands and data loading commands including performing a data format transformation, source-to-target complex scheme matching, and performing complex data value mapping wherein details of database level operations for structured data and details of text-indexing engine operations for unstructured data are hidden from a user of the source mapping commands and data loading commands;

implementing the failure recovery commands and error correction commands defining a language syntax wherein system and database level tasks required to perform the commands are transparent to a user of the failure recovery commands and error correction commands;

implementing the failure recovery commands including an abort command that uses a pending log to return the information warehouse to a consistent break point prior to a data load into the information warehouse so that operational details of maintaining data integrity for both structured and unstructured data are hidden from a user of the abort command;

implementing the failure recovery commands so that a load command automatically resumes loading data into the information warehouse from a last consistent breakpoint of a data load, while maintaining data integrity for both structured and unstructured data by performing system level tasks that are hidden from a user of the load command;

implementing the error correction commands including an undo command, operating with regard to a source file, that uses a checkpoint log to automatically return the information warehouse to a state of the information warehouse prior to loading of the source file, while maintaining data integrity for both structured and unstructured data by performing system level tasks that are hidden from a user of the undo command;

implementing the error correction commands including a redo command, operating with regard to a source file, that uses a checkpoint log to automatically return the information warehouse to a state of the information warehouse after loading of the source file, wherein the loading of the source file has previously been undone, and while maintaining data integrity for both structured and unstructured data by performing system level tasks that are hidden from a user of the redo command;

generating a graphical interface that accepts inputs from a user and generates declarative information warehouse language commands for information warehouse failure recovery and error correction;

sending user input from the graphical interface to an information warehouse language processor generating system level, database level, and indexing engine level commands for performing tasks specified by the commands from the interface;

sending system level, database level, and indexing engine level commands from the information warehouse language processor to a warehouse builder performing tasks specified by the system level, database level, and indexing engine level commands to build the IW according to the commands from the user interface, the system level, database level, and indexing engine level commands being hidden from the user; and

sending system level, database level, and indexing engine level commands from the information warehouse language to a data loader performing tasks specified by the system level, database level, and indexing engine level commands to load data from a source file into the information warehouse according to the commands from the user interface, the system level, database level, and indexing engine level commands being hidden from the user.

2-35. -35. (canceled)