Transparent archiving

• US 7,287,048 B2
• Filed: 01/07/2004
• Issued: 10/23/2007
• Est. Priority Date: 01/07/2004
• Status: Active Grant

• Current Assignee: Domo, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Wolfson, Charles Daniel, Bourbonnais, Serge
• Primary Examiner(s): Cottingham; John
• Assistant Examiner(s): Lovel; Kimberly
• Application Number:
  US10/753,905
• Publication Number:
  US 20050149584A1
• Time in Patent Office: 1,385 Days
• Field of Search: 707/204, 707/10
• US Class Current: 1/1
• CPC Class Codes: G06F 16/256 in federated or virtual databases
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

A method, apparatus, and article of manufacture implementing the method, transparently archives data. One or more data sources store data. A first data mover moves a first subset of data from the one or more data sources to a first federated archive in accordance with a first set of rules.

In another aspect of the invention, a query is processed to retrieve a second subset of data, and a portion of the second subset of data is retrieved from the first federated archive. In yet another aspect of the invention, the processing of the query determines whether any data that would satisfy the query is stored in the first federated archive.

Claims

1. A computer-implemented method of transparently archiving data stored in one or more first data sources of a federated relational database management system, comprising:

storing a first set of rules in the federated relational database management system, the first set of rules being stored in a data structure, the first set of rules comprising at least one rule, said at least one rule comprising a specified data source of said one or more first data sources, a specified archive target, a specified predicate, a specified timestamp, a specified retention period, and a specified archiving schedule, the specified predicate being a condition to evaluate, the specified retention period specifying a duration of time to retain data that satisfies the condition, the specified timestamp specifying a column of said specified data source comprising a time from which to calculate a particular retention period based on the specified retention period of the data that satisfies the condition;

reading, by a first data mover, the first set of rules that is stored in the data structure, wherein the specified data source, the specified archive target, the specified predicate, the specified timestamp, the specified retention period and the specified archiving schedule are read;

generating, by the first data mover, at least one command to move a first subset of data from the one or more first data sources to a first federated archive in accordance with the first set of rules, a particular command of said at least one command being generated to move data from the specified data source to the specified archive target based on the specified predicate, the specified timestamp and the specified retention period; and

moving, by the first data mover by causing the at least one command to be executed, the first subset of data from the one or more first data sources to the first federated archive in accordance with the first set of rules, wherein the particular command is executed in accordance with the specified archiving schedule.

2. The method of claim 1 further comprising:

moving, by the first data mover, a second subset of data from the first federated archive to a second federated archive in accordance with the first set of rules.

3. The method of claim 1 wherein the first federated archive is coupled to a second federated archive, wherein the second federated archive is cascaded with the first federated archive, further comprising:

moving, by a second data mover at the second federated archive, at least a portion of the first subset of data to the second federated archive in accordance with a second set of rules.

4. The method of claim 1 further comprising:

processing a query to retrieve a second subset of data, wherein a first portion of the second subset of data is retrieved from the first federated archive.

5. The method of claim 4 wherein said processing comprises determining whether any subset of data that would satisfy the query is stored in the first federated archive.

6. The method of claim 4 further comprising:

creating a view over at least one of the one or more first data sources and the first federated archive;

wherein said processing comprises processing the query using the view to aggregate the second subset of data from the at least one of the one or more first data sources and the first federated archive.

7. The method of claim 4 wherein said processing comprises invoking a table function to retrieve the first portion of the second subset of data from the first federated archive.

8. The method of claim 1 further comprising:

receiving a query that specifies a time range from which to retrieve data; and

processing the query to retrieve data from the first federated archive based on the time range.

9. The method of claim 1 further comprising:

receiving a query to retrieve a second subset of data;

determining whether a portion of the second subset of data requested by the query is stored in the first federated archive; and

retrieving the portion of the second subset of data from the first federated archive.

10. The method of claim 9 further comprising:

updating meta-data indicating a location of the data, and

wherein said determining whether the portion of the second subset of data requested by the query is stored in the first federated archive is based on the meta-data.

11. The method of claim 1 further comprising:

updating at Least one configuration parameter;

receiving a query; and

processing the query to retrieve a second subset of data from the first federated archive in accordance with the at least one configuration parameter.

12. The method of claim 1 wherein at least one rule from the first set of rules associates a predicate with a retention period on a table in the one or more first data sources and is combined with another rule to express a dependency between a different table in the one or more first data sources.

13. The method of claim 1 further comprising:

updating a value of a portion of the first subset of data in the first federated archive.

14. The method of claim 1 wherein the archiving schedule comprises one of performing archiving continuously and a specified archiving time.

15. An apparatus for transparently archiving data of a federated relational database management system, comprising:

a computer coupled to one or more first data sources and also to a first federated archive; and

one or more computer programs embodied on a computer readable storage medium performing;

storing a first set of rules in the federated relational database management system, the first set of rules being stored in a data structure, the first set of rules comprising at least one rule, said at least one rule comprising a specified data source of said one or more first data sources, a specified archive target, a specified predicate, a specified timestamp, a specified retention period, and a specified archiving schedule, the specified predicate being a condition to evaluate, the specified retention period specifying a duration of time to retain data that satisfies the condition, the specified timestamp specifying a column of said specified data source comprising a time from which to calculate a particular retention period based on the specified retention period of the data that satisfies the condition;

reading, by a first data mover, the first set of rules that is stored in the data structure, wherein the specified data source, the specified archive target, the specified predicate, the specified timestamp, the specified retention period and the specified archiving schedule are read;

generating, by the first data mover, at least one command to move a first subset of data from the one or more first data sources to the first federated archive in accordance with the first set of rules, a particular command of said at least one command being generated to move data from the specified data source to the specified archive target based on the specified predicate, the specified timestamp and the specified retention period; and

moving, by the first data mover by causing the at least one command to be executed, the first subset of data from the one or more first data sources to the first federated archive in accordance with the first set of rules, wherein the particular command is executed in accordance with the specified archiving schedule.

16. The apparatus of claim 15 wherein the computer is coupled to a second federated archive, and said one or more computer programs embodied on a computer readable storage medium are also performing:

moving, by the first data mover, a second subset of data from the first federated archive to the second federated archive in accordance with the first set of rules.

17. The apparatus of claim 15 wherein the first federated archive is coupled to a second federated archive, wherein the second federated archive is cascaded with the first federated archive, and said one or more computer programs embodied on a computer readable storage medium are also performing:

moving, by a second data mover at the second federated archive, at least a portion of the first subset of data to the second federated archive in accordance with a second set of rules.

18. The apparatus of claim 15 wherein said one or more computer programs embodied on a computer readable storage medium are also performing:

creating a view over at least one of the one or more first data sources and the first federated archive; and

processing a query to retrieve a second subset of data using the view, wherein a first portion of the second subset of data is retrieved from the first federated archive, and the view aggregates the second subset of data from the at least one of the one or more first data sources and the first federated archive.

19. The apparatus of claim 15 wherein the one or more computer programs embodied on a computer readable storage medium are also performing:

receiving a query to retrieve a second subset of data;

determining whether a portion of the second subset of data requested by the query is stored in the first federated archive; and

retrieving the portion of the second subset of data from the first federated archive.

20. The apparatus of claim 19 wherein the one or more computer programs embodied on a computer readable storage medium are also performing:

updating meta-data indicating a location of the data, and

wherein said determining whether the portion of the second subset of data requested by the query is stored in the first federated archive is based on the meta-data.

21. An article of manufacture comprising a computer readable storage medium embodying one or more instructions that are executable by a computer to perform transparently archiving data stored in one or more first data sources of a federated relational database management system, comprising:

storing a first set of rules in the federated relational database management system, the first set of rules being stored in a data structure, the first set of rules comprising at least one rule, said at least one rule comprising a specified data source of said one or more first data sources, a specified archive target, a specified predicate, a specified timestamp, a specified retention period, and a specified archiving schedule, the specified predicate being a condition to evaluate, the specified retention period specifying a duration of time to retain data that satisfies the condition, the specified timestamp specifying a column of said specified data source comprising a time from which to calculate a particular retention period based on the specified retention period of the data that satisfies the condition;

reading, by a first data mover, the first set of rules that is stored in the data structure, wherein the specified data source, the specified archive target, the specified predicate, the specified timestamp, the specified retention period and the specified archiving schedule are read;

generating, by the first data mover, at least one command to move a first subset of data from the one or more first data sources to a first federated archive in accordance with the first set of rules, a particular command of said at least one command being generated to move data from the specified data source to the specified archive target based on the specified predicate, the specified timestamp and the specified retention period; and

moving, by the first data mover by causing the at least one command to be executed, the first subset of data from the one or more first data sources to the specified target of the first federated archive in accordance with the first set of rules, wherein the particular command is executed in accordance with the specified archiving schedule.

22. The article of manufacture of claim 21, further comprising:

moving, by the first data mover, a second subset of data from the first federated archive to a second federated archive in accordance with the first set of rules.

23. The article of manufacture of claim 21 wherein the first federated archive is coupled to a second federated archive, wherein the second federated archive is cascaded with the first federated archive, further comprising:

moving, by a second data mover at the second federated archive, at least a portion of the first subset of data to the second federated archive in accordance with a second set of rules.

24. The article of manufacture of claim 21, further comprising:

processing a query to retrieve a second subset of data, wherein a first portion of the second subset of data is retrieved from the first federated archive.

25. The article of manufacture of claim 24 wherein said processing comprises determining whether any subset of data that would satisfy the query is stored in the first federated archive.

26. The article of manufacture of claim 24, further comprising:

creating a view over at least one of the one or more first data sources and the first federated archive,

wherein said processing comprises processing the query using the view to aggregate the second subset of data from the at least one of the one or more first data sources and the first federated archive.

27. The article of manufacture of claim 24 wherein said processing comprises invoking a table function to retrieve the first portion of the second subset of data from the first federated archive.

28. The article of manufacture of claim 27, further comprising:

updating meta-data indicating a location of the data; and

wherein said determining whether the portion of the second subset of data requested by the query is stored in the first federated archive based on the meta-data.

29. The article of manufacture of claim 21, further comprising:

receiving a query that specifies a time range from which to retrieve data; and

processing the query to retrieve data from the first federated archive based on the time range.

30. The article of manufacture of claim 21, further comprising:

receiving a query to retrieve a second subset of data;

determining whether a portion of the second subset of data requested by the query is stored in the first federated archive; and

retrieving the portion of the second subset of data from the first federated archive.

31. The article of manufacture of claim 21, further comprising:

updating at least one configuration parameter;

receiving a query, and

processing the query to retrieve a second subset of data from the first federated archive in accordance with the at least one configuration parameter.

32. The article of manufacture of claim 21 wherein at least one rule from the first set of rules associates a predicate with a retention period on a table in the one or more first data sources and is combined with another rule to express a dependency between a different table in the one or more first data sources.

33. The article of manufacture of claim 21, further comprising:

updating a value of a portion of the first subset of data in the first federated archive.