Retrieving point-in-time copies of a source database for creating virtual databases

• US 10,346,369 B2
• Filed: 03/13/2013
• Issued: 07/09/2019
• Est. Priority Date: 10/11/2012
• Status: Active Grant

• Current Assignee: Delphix Corporation (Perforce Software Incorporated)
• Original Assignee: Delphix Corporation (Perforce Software Incorporated)
• Inventors: Ahrens, Matthew Allen, Sinha, Subhadeep, Wilson, George Raymond, Biseda, Jeffrey Karr
• Primary Examiner(s): Skhoun, Hicham
• Application Number:
  US13/799,644
• Publication Number:
  US 20140108348A1
• Time in Patent Office: 2,309 Days
• Field of Search: 707639
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1456 Hardware arrangements for backup
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/065 Replication mechanisms

Abstract

A storage system maintains copies of a source database to allow virtual databases to be provisioned. Transaction logs received from the source database are applied to database blocks stored in the storage system to update the database blocks. The storage system checks if there are defect in the transaction logs of the source database. If a defect in transaction logs is detected, the storage system obtains a latest copy of the source database. The storage system matches database blocks of the received copy with database blocks already available in the storage system to determine whether a corresponding database block is already available in the storage system. If a received database block is different from database blocks already available at the storage system, the received database block is stored in the storage system. The database blocks may be matched using a hash value of the database blocks.

Claims

1. A method for storing copies of databases in a storage system, the method comprising:

maintaining, by a storage system, a plurality of copies of a source database on a storage system, each copy including a plurality of database blocks received by the storage system from a production database system, the plurality of database blocks representing the source database at a point in time, wherein one or more database blocks are associated with at least two copies, and wherein the source database is a relational database, the maintaining comprising;

receiving, by the storage system from the production database system, transaction logs of the source database;

determining, by the storage system, that the transaction logs of the source database are defective by;

determining that at least one transaction log is missing responsive to determining that sequence numbers of two consecutive transaction logs of the transaction logs differ by a value greater than one,

applying the transaction logs to a copy of the source database, and

determining that the transactions are corrupted responsive to receiving an indication of failure to process the transaction logs as result of applying the transaction logs to the copy of the source database;

responsive to determining that the transaction logs are defective, receiving, by the storage system from the production database system, a latest copy of data from the source database comprising a plurality of database blocks, and for one or more database blocks received in the latest copy;

determining, by the storage system, that the database block of the latest copy fails to match a corresponding database block of a previously stored copy of the source database, and

responsive to determining that the database block from the latest copy fails to match the corresponding database block of the previously stored copy of the source database, storing, by the storage system, the database block of the latest copy on the storage system; and

creating, by the storage system, a snapshot on the storage system representing the latest copy of the source database.

2. The method of claim 1, wherein matching the database block of the latest copy with the database block of the previously stored copy comprises comparing a hash value based on the database block of the latest copy with a hash value of the corresponding database block of the previously stored copy.

3. The method of claim 1, wherein the snapshot comprises a plurality of database blocks representing information stored in the source database at a point in time.

4. The method of claim 3, wherein the plurality of database blocks associated with the snapshot comprises one or more database blocks to which the transaction logs of the source database are applied.

5. The method of claim 1, further comprising:

responsive to receiving a request to provision a virtual database corresponding to a point in time, returning a file structure identifying database blocks comprising at least a first database block to which a first set of transaction logs is applied and second database block to which a second set of transaction logs is applied, the first and second sets of transaction logs received at different points in time.

6. The method of claim 1, wherein determining whether the transaction logs of the source database are defective comprises:

sending a request to the source database to determine whether the transaction logs are defective; and

receiving a response from the source database indicating whether the transaction logs are defective.

7. The method of claim 1, wherein the plurality of database blocks in the latest copy of the source database includes database blocks of the source database that changed since the previously stored copy of the source database was obtained as well as database blocks that did not change since the previously stored copy of the source database was obtained.

8. The method of claim 1, wherein receiving the latest copy of the source database comprises taking a full backup of the source database and restoring database blocks of the source database from the full backup.

9. The method of claim 1, wherein the transaction logs of the source database are received from a backup server configured to store a copy of the source database and receive transaction logs in a backup format from the source database, the method further comprising:

restoring the transaction logs from the backup format before applying the transaction logs.

10. A non-transitory computer-readable storage medium storing instructions for:

maintaining, by a storage system, a plurality of copies of a source database on a storage system, each copy including a plurality of database blocks received by the storage system from a production database system, the plurality of database blocks representing the source database at a point in time, wherein one or more database blocks are associated with at least two copies, and wherein the source database is a relational database, the maintaining comprising;

receiving, by the storage system from the production database system, transaction logs of the source database;

determining, by the storage system, that the transaction logs of the source database are defective by;

determining that at least one transaction log is missing responsive to determining that sequence numbers of two consecutive transaction logs of the transaction logs differ by a value greater than one,

applying the transaction logs to a copy of the source database, and

determining that the transactions are corrupted responsive to receiving an indication of failure to process the transaction logs as result of applying the transaction logs to the copy of the source database;

responsive to determining that the transaction logs are defective, receiving, by the storage system from the production database system, a latest copy of data from the source database comprising a plurality of database blocks, and for one or more database blocks received in the latest copy;

determining, by the storage system, that the database block of the latest copy fails to match a corresponding database block of a previously stored copy of the source database, and

responsive to determining that the database block from the latest copy fails to match the corresponding database block of the previously stored copy of the source database, storing, by the storage system, the database block of the latest copy on the storage system; and

creating, by the storage system, a snapshot on the storage system representing the latest copy of the source database.

11. The non-transitory computer readable storage medium of claim 10, further storing instructions for matching the database block of the latest copy with the database block of the previously stored copy by comparing a hash value based on the database block of the latest copy with a hash value of the corresponding database block of the previously stored copy.

12. The non-transitory computer readable storage medium of claim 10, further storing instructions for:

responsive to receiving a request to provision a virtual database corresponding to a point in time, returning a file structure identifying database blocks comprising at least a first database block to which a first set of transaction logs is applied and second database block to which a second set of transaction logs is applied, the first and second sets of transaction logs received at different points in time.

13. The non-transitory computer readable storage medium of claim 10, wherein the transaction logs of the source database are received from a backup server configured to store a copy of the source database and receive transaction logs in a backup format from the source database, the non-transitory computer readable storage medium further storing instructions for:

restoring the transaction logs from the backup format before applying the transaction logs.

14. A computer-implemented system for storing copies of databases in a storage system, the system comprising:

a computer processor; and

a non-transitory computer-readable storage medium storing instructions for execution by the computer processor, the instructions causing the computer processor to;

maintain, by a storage system, a plurality of copies of a source database on a storage system, each copy including a plurality of database blocks received by the storage system from a production database system, the plurality of database blocks representing the source database at a point in time, wherein one or more database blocks are associated with at least two copies, and wherein the source database is a relational database, the maintenance comprising;

receive, by the storage system from the production database system, transaction logs of the source database;

determine, by the storage system, that the transaction logs of the source database are defective by;

determining that at least one transaction log is missing responsive to determining that sequence numbers of two consecutive transaction logs of the transaction logs differ by a value greater than one,

applying the transaction logs to a copy of the source database, and

determining that the transactions are corrupted responsive to receiving an indication of failure to process the transaction logs as result of applying the transaction logs to the copy of the source database;

responsive to the determination that the transaction logs are defective, receive, by the storage system from the production database system, a latest copy of data from the source database comprising a plurality of database blocks, and

for one or more database blocks received in the latest copy;

determine, by the storage system, that the database block of the latest copy fails to match a corresponding database block of a previously stored copy of the source database, and

responsive to the determination that the database block from the latest copy fails to match the corresponding database block of the previously stored copy of the source database, store, by the storage system, the database block of the latest copy on the storage system; and

create, by the storage system, a snapshot on the storage system representing the latest copy of the source database.

15. The computer system of claim 14, wherein the instructions further cause the computer processor to match the database block of the latest copy with the database block of the previously stored copy by comparing a hash value based on the database block of the latest copy with a hash value of the corresponding database block of the previously stored copy.

16. The computer system of claim 14, wherein the instructions further cause the computer processor to:

responsive to receiving a request to provision a virtual database corresponding to a point in time, return a file structure identifying database blocks comprising at least a first database block to which a first set of transaction logs is applied and second database block to which a second set of transaction logs is applied, the first and second sets of transaction logs received at different points in time.