Method and apparatus for data integration

• US 7,143,112 B2
• Filed: 09/10/2003
• Issued: 11/28/2006
• Est. Priority Date: 09/10/2003
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Hitachi, Ltd.
• Inventors: Yagawa, Yuichi
• Primary Examiner(s): MOFIZ, APU M
• Application Number:
  US10/660,278
• Publication Number:
  US 20050055328A1
• Time in Patent Office: 1,175 Days
• Field of Search: 707/1, 707/10, 707/104.1, 707/200, 705/1
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

Aspects of the present invention provide integration of geographically distributed data. The data can be integrated in a single database. An illustrative embodiment of the invention comprises a tight combination between conventional ETL (extraction, translation, and loading) tools and conventional remote copy functionality used for data backup and recovery.

Claims

1. A computer-implemented method for collecting data from among a plurality of data sites to be stored at a central site, each data site having an associated data store, the method comprising:

providing each data site with a corresponding extraction routine;

for each data site, processing data contained in its associated data store in accordance with its corresponding extraction routine to produce first data, the corresponding extraction routine configured to store the first data in a storage location at the data site if the processing produces first data;

collecting second data which is based on first data obtained from those data sites for which their corresponding extraction routines produced said first data; and

loading all of the second data into a database,

wherein the step of collecting includes communicating the first data to a central site in accordance with a remote copy operation, receiving at the central site the first data as mirrored data, and transforming the mirrored data to produce the second data, the second data being stored at the central site,

wherein the step of providing a corresponding extraction routine comprises, for each data site;

receiving a specification information which is descriptive of data stored in a data store associated with the data site;

producing the extraction routine based on the specification information; and

communicating the extraction routine to the data site.

2. The method of claim 1 wherein the step of collecting comprises, at each data site that has first data, performing a data backup procedure on the first data, wherein the central site is viewed as a backup site.

3. The method of claim 1 wherein the step of communicating the first data includes signaling the central site at time when the remote copy operation is complete, the step of transforming the mirrored data being initiated in response to the central site being signaled.

4. The method of claim 1 wherein the step of collecting includes:

transforming the first data to produce the second data, the second data being stored at the data site;

communicating the second data to a central site in accordance with a remote copy operation, and receiving at the central site the second data as mirrored data.

5. The method of claim 1 further comprising providing each data site with a corresponding remote copy program.

6. The method of claim 1 further including providing each data site with a corresponding transformation routine, and for each data site having first data processing the first data by executing its corresponding transformation routine to produce the second data.

7. A data collection system comprising a plurality of remote data sites and a data collection site, the remote data sites and the data collection site configured to operate according to the method of claim 1.

8. A computer-implemented method for collecting data from among a plurality of data sites to be stored at a central site, each data site having an associated data store, the method comprising:

providing each data site with a corresponding extraction routine;

for each data site, processing data contained in its associated data store in accordance with its corresponding extraction routine to produce first data, the corresponding extraction routine configured to store the first data in a storage location at the data site if the processing produces first data;

collecting second data which is based on first data obtained from those data sites for which their corresponding extraction routines produced said first data, the second data being based on the first data; and

loading all of the second data into a database,

wherein the step of collecting includes communicating the first data to a central site in accordance with a remote copy operation, receiving at the central site the first data as mirrored data, and transforming the mirrored data to produce the second data, the second data being stored at the central site,

wherein the step of providing a corresponding extraction routine comprises, for each data site;

receiving a first specification information which is descriptive of data stored in its associated data store;

producing a second specification information based on the first specification information which is descriptive of the extraction routine;

communicating the second specification information to the data site; and

producing, at the data site, the extraction routine based on the second specification information.

9. The method of claim 8 wherein the second specification information is source code.

10. A data collection system comprising:

a central data site comprising at least one host processor;

a host storage system operatively coupled to the at least one host processor;

at least one remote data site comprising at least one remote processor; and

a remote storage system operatively coupled to the at least one remote processor,

the at least one host processor having program generating code configured to;

obtain storage related parameters from the at least one remote data site;

generate first interim volume managing code;

generate second interim volume managing code;

generate a data extraction routine, including receiving a specification information based on the storage related parameters and producing the extraction routine based on the specification information;

generate remote copy control code based on the storage related parameters; and

transfer the second interim volume managing code, the data extraction routine, and the remote copy control code to the at least one remote data site,

the at least one host processor further having program manager code configured to initiate processing of the second interim volume managing code, the data extraction routine, and the remote copy control code,

the data extraction routine configured to produce extracted data,

the remote copy control code configured to perform a data duplication operation of the extracted data, wherein the central data site serves as the duplication site for the extracted data,

the first interim volume managing code configured to allocate a host interim volume in the host storage system for storing at least some of the extracted data,

the second interim volume managing code configured to allocate a remote interim volume in the remote storage system for storing the extracted data.

11. The system of claim 10 wherein the data duplication operation is a remote copy operation.

12. The system of claim 10 wherein the at least some of the extracted data is copied to the central data site as the result of a data mirroring operation.

13. The system of claim 10 wherein data extraction routine includes a size calculation component to produce a data size metric indicative of the size of the extracted data, the data size metric being communicated to the first interim volume managing code and to the second interim volume managing code, wherein the host interim volume is allocated based at least on the data size metric and the remote interim volume is allocated based at least on the data size metric.

14. The system of claim 13 wherein the host interim volume has a flag associated with the remote data site, the remote copy control code further configured to set the flag to a first value when performing the backup operation and to set the flag to a second value when the backup operation has completed.