Tagging data for automatic transfer during backups

• US 10,303,555 B1
• Filed: 12/23/2017
• Issued: 05/28/2019
• Est. Priority Date: 12/23/2017
• Status: Active Grant

• Current Assignee: Rubrik, Inc.
• Original Assignee: Rubrik, Inc.
• Inventors: Lee, Looi Chow
• Primary Examiner(s): Nguyen, Than
• Application Number:
  US15/853,717
• Time in Patent Office: 521 Days
• Field of Search: 711161-162
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/167 Error detection by comparing the memory output
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 2201/83 the solution involving signatures
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

In one approach, data blocks or files that have a history of change are tagged for automatic transfer to backup on the assumption that they have changed since the last backup. Other data blocks and files are first tested for change, for example by comparing digital fingerprints of the current data versus the previously backed up data, before transferring to backup.

Claims

1. In a data management and storage (DMS) system, a method for pulling a snapshot of a fileset from a compute infrastructure serviced by the DMS system, the method comprising:

determining whether a data block in the fileset is currently tagged by the DMS system for automatic transfer, wherein whether the data block is tagged depends on the DMS system'"'"'s analysis of a change history of a portion of the fileset relating to the data block;

if the data block is currently tagged for automatic transfer, then transferring the data block from the compute infrastructure to the DMS system without first determining whether the data block is the same as a previous snapshot of the data block already stored in the DMS system;

if the data block is not currently tagged for automatic transfer, then;

transferring a digital fingerprint of the previous snapshot of the data block to the compute infrastructure;

causing the compute infrastructure to calculate a digital fingerprint of the data block and to determine whether the digital fingerprints of the data block and of the previous snapshot are the same; and

if the digital fingerprints are not the same, then transferring the data block from the compute infrastructure to the DMS system but, if the digital fingerprints are the same, then not transferring the data block.

2. The method of claim 1, wherein whether the data block is tagged depends on the DMS system'"'"'s analysis of a change history of the data block itself.

3. The method of claim 1, wherein the data block is part of a file, and whether the data block is tagged depends on the DMS system'"'"'s analysis of a change history of the file.

4. The method of claim 3, wherein whether the data block is tagged depends on the DMS system'"'"'s analysis of a change history of other data blocks contained in the file.

5. The method of claim 1, wherein the data block is part of a file, and all data blocks in the file are always tagged the same.

6. The method of claim 1, wherein the data block is part of a file, and different data blocks in the file may be tagged differently.

7. The method of claim 1, wherein the data block is part of a file, and whether the data block is tagged also depends on a file type of the file.

8. The method of claim 1, wherein the data block is part of a file, and whether the data block is tagged also depends on a file extension of the file.

9. The method of claim 1, wherein the data block is part of a file, and whether the data block is tagged also depends on contents of the file.

10. The method of claim 1, wherein the data block is part of a file, and whether the data block is tagged also depends on a size of the file.

11. The method of claim 1, wherein whether the data block is tagged depends on a change rate for the previous snapshot of the data block.

12. The method of claim 1, wherein the data block is part of a file, and whether the data block is tagged depends on a change rate for a previous snapshot of the file.

13. The method of claim 1, wherein whether the data block is tagged depends on a change history for two or more previous snapshots of the portion of the fileset relating to the data block.

14. The method of claim 1, wherein whether the data block is tagged depends on whether an estimated probability that the data block has changed since the previous snapshot is greater than 1—

(a cost of calculating the digital fingerprint of the data block and determining whether the digital fingerprints of the data block and of the previous snapshot are the same)/(a cost of transferring the data block).

15. The method of claim 1, wherein a tag for automatic transfer of the data block expires after a predetermined period.

16. The method of claim 1, wherein the digital fingerprint is a hash.

17. The method of claim 1, wherein:

the DMS system comprises a DMS cluster of peer DMS nodes, a DMS database implemented across the peer DMS nodes and a distributed data store implemented across the peer DMS nodes;

whether the data block is tagged is stored in the DMS database, and the previous snapshot is stored in the distributed data store;

jobs to transfer data blocks from the compute infrastructure to the distributed data store are posted to a job queue accessible by the peer DMS nodes; and

the peer DMS nodes autonomously fetch and execute jobs from the job queue.

18. A data management and storage (DMS) cluster comprising:

a plurality of peer DMS nodes that autonomously service a compute infrastructure;

a DMS database implemented across the peer DMS nodes;

a distributed data store implemented across the peer DMS nodes;

wherein each of the peer DMS nodes includes a software stack for, in response to a job of pulling a snapshot of a fileset;

determining whether a data block in the fileset is currently tagged by the DMS system for automatic transfer, wherein whether the data block is tagged depends on the DMS system'"'"'s analysis of a change history of a portion of the fileset relating to the data block;

if the data block is currently tagged for automatic transfer, then transferring the data block from the compute infrastructure to the DMS system without first determining whether the data block is the same as a previous snapshot of the data block already stored in the DMS system;

if the data block is not currently tagged for automatic transfer, then;

transferring a digital fingerprint of the previous snapshot of the data block to the compute infrastructure;

causing the compute infrastructure to calculate a digital fingerprint of the data block and to determine whether the digital fingerprints of the data block and of the previous snapshot are the same; and

if the digital fingerprints are not the same, then transferring the data block from the compute infrastructure to the DMS system but, if the digital fingerprints are the same, then not transferring the data block.

19. The DMS cluster of claim 18, wherein at least some of the peer DMS nodes are either (a) physical machines comprising a hardware processor, memory, and data storage in addition to the software stack;

and/or (b) virtual machines.

20. A non-transitory computer-readable medium comprising instructions that when executed by a processor cause the processor to execute a method for pulling a snapshot of a fileset of a compute infrastructure serviced by a DMS system, the method comprising:

determining whether a data block in the fileset is currently tagged by the DMS system for automatic transfer, wherein whether the data block is tagged depends on the DMS system'"'"'s analysis of a change history of a portion of the fileset relating to the data block;

if the data block is currently tagged for automatic transfer, then transferring the data block from the compute infrastructure to the DMS system without first determining whether the data block is the same as a previous snapshot of the data block already stored in the DMS system;

if the data block is not currently tagged for automatic transfer, then;

transferring a digital fingerprint of the previous snapshot of the data block to the compute infrastructure;

causing the compute infrastructure to calculate a digital fingerprint of the data block and to determine whether the digital fingerprints of the data block and of the previous snapshot are the same; and

if the digital fingerprints are not the same, then transferring the data block from the compute infrastructure to the DMS system but, if the digital fingerprints are the same, then not transferring the data block.