System and method for purging database update image files after completion of associated transactions

• US 6,553,392 B1
• Filed: 10/14/1999
• Issued: 04/22/2003
• Est. Priority Date: 02/04/1999
• Status: Expired due to Term

• Current Assignee: Hewlett Packard Enterprise Development LP (Hewlett-Packard Enterprise Company)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Whitworth, P. Simon, Mosher, Malcolm Jr.
• Primary Examiner(s): Vu, Kim
• Assistant Examiner(s): Hwang, Joon Hwan
• Application Number:
  US09/418,425
• Time in Patent Office: 1,286 Days
• Field of Search: 707/1-10, 707/200-204, 707/206, 709/248, 714/2, 714/15, 714/16, 714/20, 382/100, 382/325
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2066 Optimisation of the communication load
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 11/2089 Redundant storage control functionality
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 2201/855 Details of asynchronous mirroring using a journal to transfer not-yet-mirrored changes
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

A primary computer system has a database, application programs that modify the local database, and a transaction manager that stores audit records in a local image trail reflecting those application program modifications to the local database. In a remote backup system, a Receiver process receives audit records from the primary system. The audit records include audit update and audit backout records indicating database updates and database backouts generated by transactions executing on the primary system. The Receiver stores the audit update and audit backout records in one or more image trails. For each image trail there is an Updater process that applies to a backup database volume the database updates and backouts indicated by the audit update and audit backout records in the image trail. The remote backup system periodically executes a file purge procedure, which identifies the oldest transaction table from among the transaction tables in the last image trail file accessed for each of the image trails. Then, for each image trail, the file purge procedure accesses the image trial files in a predefined chronological order and for each accessed image trail file it compares a first set of newest transaction identifiers in the file'"'"'s transaction table with a second set of oldest transaction identifiers in the identified oldest transaction table. The procedure purges the accessed image trail file only when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

Claims

1. A method of operating a backup system so as to replicate database updates performed on a primary system, the method comprising:

receiving a stream of audit records from the primary system, the audit records including audit update records indicating database updates generated by transactions executing on the primary system;

storing the audit update records in one or more image trails, and storing each image trail in a sequence of image trail files;

storing in each image trail file a transaction table representing a range of transaction identifiers for transactions potentially pending in the primary system at the time that the first audit record in the image trail file was generated by the primary system;

for each image trail, accessing and processing the audit records in the sequence of image trail files for that image trail; and

periodically executing a file purge procedure for purging image trail files no longer needed, including;

identifying an oldest transaction table from among a set of transaction tables each of which comprises the transaction table in the last image trail file accessed for each of the image trails;

accessing an image trail file for one of the image trails;

comparing a first set of newest transaction identifiers in the transaction table in the accessed image trail file with a second set of oldest transaction identifiers in the identified oldest transaction table, and conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

2. A method of operating a backup system so as to replicate database updates performed on a primary system, the method comprising:

receiving a stream of audit records from the primary system, the audit records including audit update records indicating database updates generated by transactions executing on the primary system, transaction state records and time interval control records, at least a subset of the transaction state records indicating a commit/abort outcome for a specified transaction;

each audit update record and transaction state record including a transaction identifier that identifies a corresponding transaction on the primary system;

storing the audit update records in one or more image trails;

inspecting the received transaction state records in a predefined chronological order and generating a current transaction table representing a range of transaction identifiers for transactions for which there is at least one transaction state record between successive ones of the time interval control records in the stream of audit records;

saving the current transaction table as a previous transaction table and generating a new current transaction table whenever a time interval control record is received;

storing each of the image trails as a sequence of image trail files, including generating a new image trail file each time a previous image trail file reaches a predefined state, and storing in each new image trail file a copy of the previous transaction table at the time that the new image trail file is generated;

for each image trail, accessing and processing the audit records in the sequence of image trail files for that image trail; and

periodically executing a file purge procedure for purging image trail files no longer needed, including;

identifying an oldest transaction table copy from among a set of transaction table copies that comprises the transaction table copy in the last image trail file accessed for each of the image trails;

accessing an image trail file for one of the image trails;

comparing a first set of newest transaction identifiers in the transaction table copy in the accessed image trail file with a second set of oldest transaction identifiers in the identified oldest transaction table copy, and conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

3. The method of claim 2, wherein the step of periodically executing a file purge procedure includes:

storing in a global transaction table information including the second set of oldest transaction identifiers in the identified oldest transaction table copy;

for each image trail for which there are more than a predefined (RetainCount) number of image trail files that have not been purged, performing the steps of accessing an image trail file, comparing the first and second sets of transaction identifiers, and conditionally purging the accessed image trail file.

4. The method of claim 3, wherein the step of periodically executing a file purge procedure includes:

for each image trail for which there are more than a predefined (RetainCount) number of image trail files that have not been purged,

accessing the image trail files for the image trail in chronological order, excluding the RetainCount most recent image trail files;

for each accessed image trail file comparing the first and second sets of transaction identifiers; and

conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

5. The method of claim 4, wherein the step of periodically executing a file purge procedure includes:

for each image trail for which there are more than a predefined (RetainCount) number of image trail files that have not been purged,

accessing the image trail files for the image trail in chronological order, excluding the RetainCount most recent image trail files;

for each accessed image trail file comparing the first and second sets of transaction identifiers;

conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set; and

stopping the accessing of the image trail files for the image trail when any of the transaction identifiers in the first set are not older than corresponding transaction identifiers in the second set.

6. A computer program product for use in conjunction with a backup computer system so as to replicate database updates performed on a primary system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising:

a Receiver Module that receives and stores in one or more image trails a stream of audit records received from the primary system, the audit records including audit update records indicating database updates generated by transactions executing on the primary system, and stores each image trail in a sequence of image trail files;

the Receiver Module storing in each image trail file a transaction table representing a range of transaction identifiers for transactions potentially pending in the primary system at the time that the first audit record in the image trail file was generated by the primary system;

at least one Updater Module that sequentially applies to a backup database the database updates indicated by the audit update records, in the order the audit update records are stored in the image trails; and

a file purge procedure for purging image trail files no longer needed, the file purge procedure including instructions for;

identifying an oldest transaction table from among a set of transaction tables that comprises the transaction table in the last image trail file accessed for each of the image trails;

accessing an image trail file for one of the image trails;

comparing a first set of newest transaction identifiers in the transaction table in the accessed image trail file with a second set of oldest transaction identifiers in the identified oldest transaction table, and conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

7. A method of operating a backup system so as to replicate database updates performed on a primary system, the method comprising:

a Receiver Module that receives and stores in one or more image trails a stream of audit records received from the primary system, the audit records including audit update records indicating database updates generated by transactions executing on the primary system, transaction state records and time interval control records, at least a subset of the transaction state records indicating a commit/abort outcome for a specified transaction;

each audit update record and transaction state record including a transaction identifier that identifies a corresponding transaction on the primary system;

the Receiver Module storing the audit update records in one or more image trails;

the Receiver Module includes instructions for;

inspecting the received transaction state records in a predefined chronological order and generating a current transaction table representing a range of transaction identifiers for transactions for which there is at least one transaction state record between successive ones of the time interval control records in the stream of audit records;

saving the current transaction table as a previous transaction table and generating a new current transaction table whenever a time interval control record is received;

storing each of the image trails as a sequence of image trail files, including generating a new image trail file each time a previous image trail file reaches a predefined state, and storing in each new image trail file a copy of the previous transaction table at the time that the new image trail file is generated;

at least one Updater Module that sequentially applies to a backup database the database updates indicated by the audit update records, in the order the audit update records are stored in the image trails; and

a file purge procedure for purging image trail files no longer needed, the file purge procedure including instructions for;

identifying an oldest transaction table copy from among a set of transaction table copies, each of which comprises the transaction table copy in the last image trail file accessed for each of the image trails;

accessing an image trail file for one of the image trails;

comparing a first set of newest transaction identifiers in the transaction table copy in the accessed image trail file with a second set of oldest transaction identifiers in the identified oldest transaction table copy, and conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

8. The computer program product of claim 7, wherein the file purge procedure includes instructions for:

storing in a global transaction table information including the second set of oldest transaction identifiers in the identified oldest transaction table copy;

for each image trail for which there are more than a predefined (RetainCount) number of image trail files that have not been purged, performing the steps of accessing an image trail file, comparing the first and second sets of transaction identifiers, and conditionally purging the accessed image trail file.

9. The computer program product of claim 8, wherein the file purge procedure includes instructions for:

for each image trail for which there are more than a predefined (RetainCount) number of image trail files that have not been purged,

accessing the image trail files for the image trail in chronological order, excluding the RetainCount most recent image trail files;

for each accessed image trail file comparing the first and second sets of transaction identifiers; and

conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

10. The computer program product of claim 9, wherein the file purge procedure includes instructions for:

for each image trail for which there are more than a predefined (RetainCount) number of image trail files that have not been purged,

accessing the image trail files for the image trail in chronological order, excluding the RetainCount most recent image trail files;

for each accessed image trail file comparing the first and second sets of transaction identifiers;

conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set; and

stopping the accessing of the image trail files for the image trail when any of the transaction identifiers in the first set are not older than corresponding transaction identifiers in the second set.

11. A backup computer system for replicating database updates performed on a primary system, comprising:

a backup database;

a Receiver Module for processing a stream of audit records received from the primary system, the audit records including audit update records indicating database updates generated by transactions executing on the primary system, and stores each image trail in a sequence of image trail files;

the Receiver Module storing in each image trail file a transaction table representing a range of transaction identifiers for transactions potentially pending in the primary system at the time that the first audit record in the image trail file was generated by the primary system;

at least one Updater Module that sequentially applies to a backup database the database updates indicated by the audit update records, in the order the audit update records are stored in the image trails; and

a file purge procedure for purging image trail files no longer needed, the file purge procedure including instructions for;

identifying an oldest transaction table from among a set of transaction tables, each of which comprises the transaction table in the last image trail file accessed for each of the image trails;

accessing an image trail file for one of the image trails;

comparing a first set of newest transaction identifiers in the transaction table in the accessed image trail file with a second set of oldest transaction identifiers in the identified oldest transaction table, and conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

12. A backup computer system for replicating database updates performed on a primary system, comprising:

a backup database;

a Receiver Module that receives and stores in one or more image trails a stream of audit records received from the primary system, the audit records including audit update records indicating database updates generated by transactions executing on the primary system, transaction state records and time interval control records, at least a subset of the transaction state records indicating a commit/abort outcome for a specified transaction;

each audit update record and transaction state record including a transaction identifier that identifies a corresponding transaction on the primary system;

the Receiver Module storing the audit update records in one or more image trails;

the Receiver Module includes instructions for;

inspecting the received transaction state records in a predefined chronological order and generating a current transaction table representing a range of transaction identifiers for transactions for which there is at least one transaction state record between successive ones of the time interval control records in the stream of audit records;

saving the current transaction table as a previous transaction table and generating a new current transaction table whenever a time interval control record is received;

storing each of the image trails as a sequence of image trail files, including generating a new image trail file each time a previous image trail file reaches a predefined state, and storing in each new image trail file a copy of the previous transaction table at the time that the new image trail file is generated;

at least one Updater Module that sequentially applies to a backup database the database updates indicated by the audit update records, in the order the audit update records are stored in the image trails; and

a file purge procedure for purging image trail files no longer needed, the file purge procedure including instructions for;

identifying an oldest transaction table copy from among a set of transaction table copies, each of which comprises the transaction table copy in the last image trail file accessed for each of the image trails;

accessing an image trail file for one of the image trails;

comparing a first set of newest transaction identifiers in the transaction table copy in the accessed image trail file with a second set of oldest transaction identifiers in the identified oldest transaction table copy, and conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

13. The backup computer system of claim 12, wherein the file purge procedure includes instructions for:

storing in a global transaction table information including the second set of oldest transaction identifiers in the identified oldest transaction table copy;

for each image trail for which there are more than a predefined (RetainCount) number of image trail files that have not been purged, performing the steps of accessing an image trail file, comparing the first and second sets of transaction identifiers, and conditionally purging the accessed image trail file.

14. The backup computer system of claim 13, wherein the file purge procedure includes instructions for:

for each image trail for which there are more than a predefined (RetainCount) number of image trail files that have not been purged,

accessing the image trail files for the image trail in chronological order, excluding the RetainCount most recent image trail files;

for each accessed image trail file comparing the first and second sets of transaction identifiers; and

conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

15. The backup computer system of claim 14, wherein the file purge procedure includes instructions for:

for each image trail for which there are more than a predefined (RetainCount) number of image trail files that have not been purged,

accessing the image trail files for the image trail in chronological order, excluding the RetainCount most recent image trail files;

for each accessed image trail file comparing the first and second sets of transaction identifiers;

conditionally purging the accessed image trail file when all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set; and

stopping the accessing of the image trail files for the image trail when any of the transaction identifiers in the first set are not older than corresponding transaction identifiers in the second set.