Method, system, and apparatus for providing resilient data transfer in a data protection system

• US 20060053121A1
• Filed: 09/09/2004
• Published: 03/09/2006
• Est. Priority Date: 09/09/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zizys, Giedrius, Patiejunas, Kestutis
• Granted Patent: US 8,145,601 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning

Abstract

A method for transmitting records of changes to data from a production location to a storage location is provided. The method stores in a log, records of changes to data stored at a production location. Those records are transmitted and a transmitted records checkpoint is generated that is transmitted at the end of the transmitted records. A records checkpoint confirmation is received and the plurality of transmitted records are purged from the log.

Claims

1. A method for transmitting records of changes to data from a production location to a storage location, comprising:

storing in a log, records of changes to data stored at a production location;

transmitting a plurality of the stored records;

generating a transmitted records checkpoint;

subsequent to transmitting the plurality of stored records, transmitting the generated transmitted records checkpoint;

receiving a transmitted records checkpoint confirmation; and

removing the plurality of transmitted records from the log.

2. The method of claim 1, wherein the checkpoint includes an identification of a location within the log of a last record that was transmitted with the plurality of records.

3. The method of claim 1, wherein subsequent to transmitting a plurality of the stored records, additional records of changes to data are stored in the log.

4. The method of claim 1, wherein the received transmitted records checkpoint confirmation indicates that each of the plurality of the stored records transmitted prior to the transmission of the transmitted records checkpoint have been received at a storage location.

5. The method of claim 1, wherein the received transmitted records checkpoint is received from a data management system.

6. The method of claim 1, wherein the plurality of the records includes metadata records about the data stored at the production location.

7. A method for receiving and applying changes to a replica, comprising:

receiving a plurality of records;

applying at least a portion of the received records to the replica;

subsequent to applying at least a portion of the plurality of received records, generating an applied checkpoint;

transmitting the applied checkpoint;

receiving an applied checkpoint confirmation; and

purging the applied records.

8. The method of claim 7, further comprising:

storing the plurality of received records in a log; and

subsequent to receiving the applied checkpoint confirmation, purging from the log the applied records.

9. The method of claim 7, further comprising:

for each received record, determining if the record is a change to be made to the replica; and

in response to a determination that the record is a change to be made to the replica, applying the record to the replica.

10. The method of claim 7, wherein the replica is a copy of data stored at a production location.

11. The method of claim 7, wherein the generated applied checkpoint includes an identification of a location in a log of the last applied record.

12. The method of claim 7, wherein the received applied checkpoint confirmation is received from a data management system.

13. The method of claim 7, further comprising:

in response to a determination that the record is not a change to be made to the replica,

determining if the record is a checkpoint; and

in response to a determination that the record is a checkpoint, adding a log identification to the checkpoint and transmitting the checkpoint.

14. The method of claim 13, wherein the checkpoint is transmitted to a data protection system.

15. The method of claim 7, further comprising:

in response to a determination that the record is not a change to be made to the replica, determining if the record is a checkpoint; and

in response to a determination that the record is a checkpoint, transmitting the checkpoint and a difference log.

16. The method of claim 15, wherein the checkpoint and the difference log are transmitted to a data protection system.

17. The method of claim 15, wherein the difference log includes a list of items of data stored at a storage location that are different than items of data stored at a production location.

18. The method of claim 7, further comprising;

in response to a determination that the record is not a change to be made to the replica,

generating metadata about a portion of the replica;

comparing the generated metadata with the record; and

if the generated metadata does not match the record, adding to a difference log an identification of the portion of the replica.

19. In a data protection system having a production location, a storage location, and a data protector, a method for restarting a transmission and application of changed data between the production location and the storage location, the method comprising:

determining a first log point from which the storage location last applied a record;

transmitting the determined first log point with a command to restart applying records from the first determined log point;

determining a second log point from which the production location last transmitted a record that was received by the storage location; and

transmitting the determined second log point with a command to restart transmission of records from the determined second log point.

20. The data protection system of claim 19, wherein the first log point is determined based on a last applied checkpoint received by the data management system.

21. The data protection system of claim 20, wherein the received checkpoint includes an identification of a log point of a last transmitted record that was received by the storage location.

22. The data protection system of claim 20, wherein the applied checkpoint includes an identification of a log point of a last applied record.

23. The data protection system of claim 19, further comprising:

determining a third log point from which the storage location last received a transmitted checkpoint; and

transmitting the determined third log point with a command to begin storing records subsequent to the third log point.

24. The data protection system of claim 19, wherein the second log point is determined based on a last transmitted checkpoint received by the data protector.

25. The data protection system of claim 19, further comprising:

determining if a validation process was in progress; and

in response to determining that validation was in progress, restarting validation.

26. In a data protection system having a production location, a storage location, and a data protector, a method for restarting a validation of data process between data stored on a production location and data stored on a storage location, the method comprising:

transmitting a command to restart application of records at the storage location;

transmitting a command to restart transmission of records from the production location;

generating a marker;

transmitting the marker to the production location;

receiving the marker from the storage location; and

transmitting a command to restart validation.

27. The data protection system of claim 26, wherein restarting validation comprises:

determining a data location from which validation is to resume; and

transmitting the determined data location with a command to restart validation from the determined location.

28. The data protection system of claim 27, wherein the data location is determined from a checkpoint received from the storage location.

29. The data protection system of claim 26, wherein the marker is unique.

30. The data protection system of claim 26, wherein the marker is received by the production location and transmitted to the storage location.

31. A system for transferring data from a production location to a storage location, comprising:

a data protector configured to monitor and control a transfer of data from the production location to the storage location;

a first log configured to maintain a plurality of records of changes made to data at the production location, wherein the records are transmitted at a predetermined time from the production location to the storage location;

a first checkpoint generation device located at the production location, wherein the first checkpoint generation device generates a first checkpoint that is included with the transmitted records;

a second log configured to maintain a plurality of records received from the production location; and

a second checkpoint generation device located at the storage location, wherein the second checkpoint generation device generates a second checkpoint that is forwarded to the data protector.

32. The system of claim 31, wherein the data protector maintains a database of first and second checkpoints.

33. The system of claim 31, wherein the first checkpoint includes an identification of a first log point of a last transmitted record.

34. The system of claim 31, wherein the storage location adds a second log point to a received first checkpoint, and transmits the first checkpoint to the data protector.

35. The system of claim 31, wherein the data protector monitors the transmission of data between the storage location and the production location by receiving first and second checkpoints from the storage location.

36. The system of claim 31, wherein the data protector initiates validation of data between the production location and the storage location.