System and method for purging database update image files after completion of associated transactions for a database replication system with multiple audit logs

• US 20020194204A1
• Filed: 06/15/2001
• Published: 12/19/2002
• Est. Priority Date: 06/15/2001
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Mosher, Malcolm Jr., Whitworth, Peter Simon
• Granted Patent: US 6,691,115 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/1474 in transactions G06F16/20 takes precedence
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

A method and system for purging database update image files after completion of associated transactions for a database replication system with multiple audit logs. Audit update records and audit backout records are generated by the primary system, and are transmitted to the backup system in multiple streams in parallel. The backup system stores the received audit records as audit image trails, and applies the audit updates and audit backouts to the backup database without regard to whether the associated transactions committed or aborted. Because audit updates and audit backouts are applied without regard to whether the associated transactions committed or aborted, image files containing audit records associated with transactions that have not yet committed or aborted must not be purged. The present invention provides for a method for determining whether an image trail file contains audit records that can be purged such that image trail files can be safely deleted.

Claims

1. A method of operating a backup system so as to replicate database updates performed on a primary system, the method comprising:

receiving a first stream of first audit records from the primary system, the first audit records including first audit update records indicating database updates generated by transactions executing on the primary system and control point records defining control time frames at which the first audit update records were generated by the primary system;

upon receiving each of the control point records, generating a system transaction record representing a range of transaction identifiers for transactions executed in the primary system during the control time frame associated with the control point record, the system transaction record further having a control point count representative of the control time frame;

storing the first audit update records in one or more first image trails, and storing each first image trail in first image trail files;

storing in each first image trail file a control point count for identifying the control time frame at which the first image trail file is created;

concurrently for each of the first image trails, applying the first audit records to a backup database in the sequence of the first image trail files; and

periodically executing a file purge procedure for purging image trail files no longer needed, including;

building a composite system transaction record to identify a maximum span of transaction identifiers corresponding to control time frames associated with the first audit records currently applied to the backup database,

selecting one of the first image trail files and identifying the control point count stored in the selected first image trail file;

retrieving a first one of the system transaction records that corresponds to the identified control point count; and

comparing a first set of transaction identifiers in the first system transaction record to second set of transaction identifiers in the composite system transaction record, and purging the accessed first image trail file provided all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

2. The method of claim 1, wherein the building step comprises:

determining a set of the system transaction records that are generated since a previous execution of the file purge procedure; and

storing a lowest transaction identifier and a highest transaction identifier of the set of the system transaction records in the composite system transaction record.

3. The method of claim 1, further comprising:

receiving a second stream of second audit records, the second audit records including audit update records indicating database updates generated by transactions executing on the primary system;

storing the second audit records in one or more second image trails, and storing each of the second image trails in second image trail files;

for each of the second image trails, storing in each second image trail file an audit trail position associated with a last audit update record stored in a previous one of the second image files of the same second image trail, the audit trail position being associated with one of the control time frames;

concurrently for each second image trail, applying the second audit records to the backup database in the sequence of second image trail files; and

the file purge procedure purging second image trail files no longer needed, including;

constructing a conversion table that indicates, for each of the control point counts, a range of second audit records associated with each of the control point counts;

accessing one of the second image trail file and identifying an audit trail position stored within the accessed second image trail file;

looking up the conversion table to determine a control point count associated with the retrieved audit trail position;

retrieving a second one of the system transaction records associated with the determined control point count; and

comparing a third set of transaction identifiers in the second system transaction record to the second set of transaction identifiers in the composite system transaction record, and purging the accessed second image trail file provided that all of the transaction identifiers in the third set are older than corresponding transaction identifiers in the second set.

4. The method of claim 3, wherein the step of periodically executing a file purge procedure includes:

for each first image trail for which there are more than a predefined number of first image trail files that have not been processed, performing the steps of accessing a first image trail file, comparing the first and second sets of transaction identifiers, and conditionally purging the accessed first image trail file; and

for each second image trail for which there are more than the predefined number of second image trail files that have not been processed, performing the steps of accessing a second image trail file, comparing the second and third sets of transaction identifiers, and conditionally purging the accessed second image trail file.

5. The method of claim 3, wherein the step of periodically executing a file purge procedure includes:

for each first image trail for which there are more than a predefined number of first image trail files that have not been processed,

accessing the first image trail files in reverse chronological order, excluding the predefined number of most recent first image trail files;

for each accessed first image trail file comparing the first and second sets of transaction identifiers; and

purging the accessed first image trail file provided all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

6. The method of claim 5, wherein the step of periodically executing a file purge procedure further comprises:

for each second image trail for which there are more than the predefined number of second image trail files that have not been processed,

accessing the second image trail files in reverse chronological order, excluding the predefined number of most recent second image trail files;

for each accessed second image trail file comparing the second and third sets of transaction identifiers; and

purging the accessed second image trail file provided all of the transaction identifiers in the third set are older than corresponding transaction identifiers in the second set.

7. A computer program product for use in conjunction with a backup computer system so as to replicate database updates performed on a primary system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising:

a Master Receiver Module that receives and stores in one or more first image trails a stream of first audit records received from the primary system, the first audit records including audit update records indicating database updates generated by transactions executing on the primary system and control point records defining control time frames at which the first audit update records were generated by the primary system;

the Master Receiver Module, upon receiving each of the control point records, generating a system transaction record representing a range of transaction identifiers for transactions executed in the primary system during the control time frame associated with the control point record, each system transaction record further having a control point count representative of the control time frame;

the Master Receiver Module storing each first image trail in a sequence of first image trail files and storing in each first image trail file a control point count associated with a last audit record stored in a previous one of the first image trail files;

an Updater Module for each first image trail that sequentially applies to a backup database the database updates indicated by the audit update records, in the order the audit update records are stored in the first image trails; and

a Purger Module for periodically executing a file purge procedure for purging image trail files no longer needed, the Purger Module including instructions for;

building a composite system transaction record to identify a maximum span of transaction identifiers corresponding to control time frames associated with the first audit records currently applied to the backup database,

selecting one of the first image trail files and identifying the control point count stored in the selected first image trail file;

retrieving a first one of the system transaction records that corresponds to the identified control point count; and

comparing a first set of transaction identifiers in the first system transaction record to second set of transaction identifiers in the composite system transaction record, and purging the accessed first image trail file provided all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

8. The computer program product of claim 7, wherein the Purger module further comprises instructions for:

determining a set of the system transaction records that are generated since a previous execution of the file purge procedure; and

storing a lowest transaction identifier and a highest transaction identifier of the set of the system transaction records in the composite system transaction record.

9. The computer program product of claim 7, further comprising:

a Receiver Module that receives and stores in one or more second image trails a stream of second audit records received from the primary system, the second audit records including audit update records indicating database updates generated by transactions executing on the primary system;

the Receiver Module storing each second image trail in a sequence of second image trail files and storing in each second image trail file an audit trail position associated with a last audit update record stored in a previous one of the second image files;

an Updater Module for each second image trail that sequentially applies to a backup database the database updates indicated by the audit update records, in the order the audit update records are stored in the second image trails; and

wherein the Purger Module further comprises instructions for;

constructing a conversion table that indicates, for each of the control point counts, a range of second audit records associated with each of the control point counts;

accessing one of the second image trail file and identifying an audit trail position stored within the accessed second image trail file;

looking up the conversion table to determine a control point count associated with the retrieved audit trail position;

retrieving a second one of the system transaction records associated with the determined control point count; and

comparing a third set of transaction identifiers in the second system transaction record to the second set of transaction identifiers in the composite system transaction record, and purging the accessed second image trail file provided that all of the transaction identifiers in the third set are older than corresponding transaction identifiers in the second set.

10. The computer program product of claim 9, wherein Purger Module further includes instructions for:

for each first image trail for which there are more than a predefined number of first image trail files that have not been processed, performing the steps of accessing a first image trail file, comparing the first and second sets of transaction identifiers, and conditionally purging the accessed first image trail file; and

for each second image trail for which there are more than the predefined number of second image trail files that have not been processed, performing the steps of accessing a second image trail file, comparing the second and third sets of transaction identifiers, and conditionally purging the accessed second image trail file.

11. The computer program product of claim 9, wherein Purger Module further includes instructions for:

for each first image trail for which there are more than a predefined number of first image trail files that have not been processed,

accessing the first image trail files in reverse chronological order, excluding the predefined number of most recent first image trail files;

for each accessed first image trail file comparing the first and second sets of transaction identifiers; and

purging the accessed first image trail file provided all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

12. The computer program product of claim 11, wherein Purger Module further includes instructions for:

for each second image trail for which there are more than the predefined number of second image trail files that have not been processed,

accessing the second image trail files in reverse chronological order, excluding the predefined number of most recent second image trail files;

for each accessed second image trail file comparing the second and third sets of transaction identifiers; and

purging the accessed second image trail file provided all of the transaction identifiers in the third set are older than corresponding transaction identifiers in the second set.

14. The backup computer system of claim 13, wherein the Purger module further comprises instructions for:

determining a set of the system transaction records that are generated since a previous execution of the file purge procedure; and

storing a lowest transaction identifier and a highest transaction identifier of the set of the system transaction records in the composite system transaction record.

15. The backup computer system of claim 13, further comprising:

a Receiver Module that receives and stores in one or more second image trails a stream of second audit records received from the primary system, the second audit records including audit update records indicating database updates generated by transactions executing on the primary system;

the Receiver Module storing each second image trail in a sequence of second image trail files and storing in each second image trail file an audit trail position associated with a last audit update record stored in a previous one of the second image files;

an Updater Module for each second image trail that sequentially applies to a backup database the database updates indicated by the audit update records, in the order the audit update records are stored in the second image trails; and

wherein the Purger Module further comprises instructions for;

constructing a conversion table that indicates, for each of the control point counts, a range of second audit records associated with each of the control point counts;

accessing one of the second image trail file and identifying an audit trail position stored within the accessed second image trail file;

looking up the conversion table to determine a control point count associated with the retrieved audit trail position;

retrieving a second one of the system transaction records associated with the determined control point count; and

comparing a third set of transaction identifiers in the second system transaction record to the second set of transaction identifiers in the composite system transaction record, and purging the accessed second image trail file provided that all of the transaction identifiers in the third set are older than corresponding transaction identifiers in the second set.

16. The backup computer system of claim 15, wherein Purger Module further includes instructions for:

for each first image trail for which there are more than a predefined number of first image trail files that have not been processed, performing the steps of accessing a first image trail file, comparing the first and second sets of transaction identifiers, and conditionally purging the accessed first image trail file; and

for each second image trail for which there are more than the predefined number of second image trail files that have not been processed, performing the steps of accessing a second image trail file, comparing the second and third sets of transaction identifiers, and conditionally purging the accessed second image trail file.

17. The backup computer system of claim 15, wherein Purger Module further includes instructions for:

for each first image trail for which there are more than a predefined number of first image trail files that have not been processed,

accessing the first image trail files in reverse chronological order, excluding the predefined number of most recent first image trail files;

for each accessed first image trail file comparing the first and second sets of transaction identifiers; and

purging the accessed first image trail file provided all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.

18. The backup computer system of claim 17, wherein Purger Module further includes instructions for:

for each second image trail for which there are more than the predefined number of second image trail files that have not been processed,

accessing the second image trail files in reverse chronological order, excluding the predefined number of most recent second image trail files;

for each accessed second image trail file comparing the second and third sets of transaction identifiers; and

purging the accessed second image trail file provided all of the transaction identifiers in the third set are older than corresponding transaction identifiers in the second set.

13. A backup computer system for replicating database updates performed on a primary system, comprising:

a backup database;

a Master Receiver Module that receives and stores in one or more first image trails a stream of first audit records received from the primary system, the first audit records including audit update records indicating database updates generated by transactions executing on the primary system and control point records defining control time frames at which the first audit update records were generated by the primary system;

the Master Receiver Module, upon receiving each of the control point records, generating a system transaction record representing a range of transaction identifiers for transactions executed in the primary system during the control time frame associated with the control point record, each system transaction record further having a control point count representative of the control time frame;

the Master Receiver Module storing each first image trail in a sequence of first image trail files and storing in each first image trail file a control point count associated with a last audit record stored in a previous one of the first image trail files;

an Updater Module for each first image trail that sequentially applies to the backup database the database updates indicated by the audit update records, in the order the audit update records are stored in the first image trails; and

a Purger Module for periodically executing a file purge procedure for purging image trail files no longer needed, the Purger Module including instructions for;

building a composite system transaction record to identify a maximum span of transaction identifiers corresponding to control time frames associated with the first audit records currently applied to the backup database,

selecting one of the first image trail files and identifying the control point count stored in the selected first image trail file;

retrieving a first one of the system transaction records that corresponds to the identified control point count; and

comparing a first set of transaction identifiers in the first system transaction record to second set of transaction identifiers in the composite system transaction record, and purging the accessed first image trail file provided all of the transaction identifiers in the first set are older than corresponding transaction identifiers in the second set.