Synchronization of storage using log files and snapshots

• US 8,782,003 B1
• Filed: 05/13/2011
• Issued: 07/15/2014
• Est. Priority Date: 05/13/2011
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Patterson, R. Hugo
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Hershley, Mark E
• Application Number:
  US13/107,791
• Time in Patent Office: 1,159 Days
• Field of Search: 707/624
• US Class Current: 707/624
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs

Abstract

A system for processing data comprises a deduplicating system, an interface, and a processor. The deduplicating system stores a copy of data stored in a data storage system by storing a set of segments that is able to reconstruct the data stored in the data storage system. The data storage system has a stored log file. The stored log file stores a data change and an associated time for the data change. The interface receives an indication to revert data stored in the data storage system to a state at a snapshot time. The processor is configured to determine a first subset of the data stored in the data storage system that has changed since a prior snapshot using the stored log file and to determine a second subset of the data stored in the data storage system that has changed between the prior snapshot and the snapshot time using a first list of fingerprints associated with the prior snapshot and a second list of fingerprints associated with the snapshot time.

Claims

1. A system for processing data, comprising:

a deduplicating system that stores a copy of at least a portion of data stored in a data storage system at a first snapshot time at least in part by generating a first snapshot of the data, wherein the deduplicating system includes a fingerprint index that comprises a list of fingerprints associated with every unique segment stored in the deduplicating system, wherein the data storage system includes a stored log file that stores one or more data changes and times associated with each data change, and wherein the deduplicating system generates the first snapshot by;

breaking at least a portion of the data at the data storage system into a plurality of segments;

computing a fingerprint for each of at least a subset of the plurality of segments;

storing the fingerprints in the first snapshot, wherein a snapshot includes a list of fingerprints of data comprising at least the portion of the data stored in the data storage system, and wherein a fingerprint that correspond to identical segments is repeated in a snapshot;

identifying, based at least in part on the fingerprint index, fingerprints in the snapshot that are not in the fingerprint index; and

storing only segments that correspond to the identified fingerprints such that each stored segment is able to be used to reconstruct the data stored in the data storage system;

an interface for receiving an indication to revert data stored in the data storage system to a state at a snapshot time; and

a processor configured to;

determine a first subset of the data stored in the data storage system that has changed since a prior snapshot using the stored log file; and

determine a second subset of the data stored in the data storage system that has changed between the prior snapshot and the snapshot time using a first list of fingerprints associated with the prior snapshot and a second list of fingerprints associated with the snapshot time.

2. A system as in claim 1, wherein the data storage system for storing data comprises a block storage system.

3. A system as in claim 1, wherein the first subset of data comprises a list of segments.

4. A system as in claim 1, wherein the second subset of the data comprises a list of segments.

5. A system as in claim 1, wherein the first subset and the second subset are merged to create a merged subset.

6. A system as in claim 5, wherein the processor is further configured to indicate the merged subset to the data storage system.

7. A system as in claim 6, wherein the processor is further configured to transfer data associated with the merged subset from the deduplicating system to the data storage system.

8. A system as in claim 7, wherein transferring data comprises transferring data requested by a user.

9. A system as in claim 7, wherein transferring data comprises transferring all data associated with the merged subset.

10. A method for processing data, comprising:

receiving an indication to revert data stored in a data storage system to a state at a snapshot time, wherein the data storage system for storing data has a stored log file, wherein the stored log file stores a data change and an associated time for the data change, wherein a deduplicating system stores a copy of data stored in the data storage system at a first snapshot time at least in part by generating a first snapshot of the data, wherein the deduplicating system includes a fingerprint index that comprises a list of fingerprints associated with every unique segment stored in the deduplicating system, and wherein the deduplicating system generates the first snapshot by;

breaking at least a portion of the data at the data storage system into a plurality of segments;

computing a fingerprint for each of at least a subset of the plurality of segments;

storing the fingerprints in the first snapshot, wherein a snapshot includes a list of fingerprints of data comprising at least the portion of the data stored in the data storage system, and wherein a fingerprint that correspond to identical segments is repeated in a snapshot;

identifying, based at least in part on the fingerprint index, fingerprints in the snapshot that are not in the fingerprint index; and

storing only segments that correspond to the identified fingerprints such that each stored segment is able to be used to reconstruct the data stored in the data storage system;

determining, using a processor, a first subset of the data stored in the data storage system that has changed since a prior snapshot using the stored log file; and

determining a second subset of the data stored in the data storage system that has changed between the prior snapshot and the snapshot time using a first list of fingerprints associated with the prior snapshot and a second list of fingerprints associated with the snapshot time.

11. A method as in claim 10, wherein the data storage system for storing data comprises a block storage system.

12. A method as in claim 10, wherein the first subset of data comprises a list of segments.

13. A method as in claim 10, wherein the second subset of the data comprises a list of segments.

14. A method as in claim 10, wherein the first subset and the second subset are merged to create a merged subset.

15. A method as in claim 14, further comprising indicating the merged subset to the data storage system.

16. A method as in claim 15, further comprising transferring data associated with the merged subset from the deduplicating system to the data storage system.

17. A method as in claim 16, wherein transferring data comprises transferring data requested by a user.

18. A method as in claim 17, wherein transferring data comprises transferring all data associated with the merged subset.

19. A computer program product for processing data, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving an indication to revert data stored in a data storage system to a state at a snapshot time, wherein the data storage system for storing data has a stored log file, wherein the stored log file stores a data change and an associated time for the data change, wherein a deduplicating system stores a copy of data stored in the data storage system at a first snapshot time at least in part by generating a first snapshot of the data, wherein the deduplicating system includes a fingerprint index that comprises a list of fingerprints associated with every unique segment stored in the deduplicating system, and wherein the deduplicating system generates the first snapshot by;

breaking at least a portion of the data at the data storage system into a plurality of segments;

computing a fingerprint for each of at least a subset of the plurality of segments;

storing the fingerprints in the first snapshot, wherein a snapshot includes a list of fingerprints of data comprising at least the portion of the data stored in the data storage system, and wherein a fingerprint that correspond to identical segments is repeated in a snapshot;

identifying, based at least in part on the fingerprint index, fingerprints in the snapshot that are not in the fingerprint index; and

storing only segments that correspond to the identified fingerprints such that each stored segment is able to be used to reconstruct the data stored in the data storage system;

determining a first subset of the data stored in the data storage system that has changed since a prior snapshot using the stored log file; and

determining a second subset of the data stored in the data storage system that has changed between the prior snapshot and the snapshot time using a first list of fingerprints associated with the prior snapshot and a second list of fingerprints associated with the snapshot time.