Systems and methods for minimizing network bandwidth for replication/back up

• US 9,015,122 B2
• Filed: 12/21/2012
• Issued: 04/21/2015
• Est. Priority Date: 12/21/2012
• Status: Active Grant

• Current Assignee: Zetta LLC
• Original Assignee: Zetta Incorporated
• Inventors: Harrison, Jason, Montulli, Louis, Whitehead, Jeffrey, Eckhardt, Andrew, Ramachandra, Shylaja
• Primary Examiner(s): NGUYEN, CAM LINH T
• Application Number:
  US13/725,597
• Publication Number:
  US 20140181034A1
• Time in Patent Office: 851 Days
• Field of Search: 707/610, 707/639, 707/649, 707/655, 707/646
• US Class Current: 707/646
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/174 Redundancy elimination performed by the file system management of the data involved in backup or backup restore using de-duplication of the data G06F11/14
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

Systems and methods are disclosed for backing up a client data set on a computer by identifying changed files; generating a first sub-file signature for one or more segments of a changed file; subsequently generating a second sub-file signature for each file segment to detect insertions, removals, and rearrangements of data in the file; transmitting one more changed segment(s) in the file; and writing a manifest file reflecting the new target state.

Claims

1. A method for backing up a client data set on a computer, comprising:

identifying changed files by;

generating a first sub-file signature for one or more segments of a changed file; and

subsequently generating another sub-file signature for each file segment and comparing them to detect insertions, removals, or rearrangements of data in the file and storing a plurality of signatures for each file locally on the computer representing the latest state of each file uploaded to a server; and

transmitting one more changed segment(s) in the file;

writing a manifest file reflecting the new target state as an incremental backup;

compressing, encrypting and transmitted the changed segment through a Web-based Distributed Authoring and Versioning application program interface;

expanding one or more patch sets and saving the patch sets with the file system to provide an incremental backup while generating a full backup;

taking a snapshot after a full backup providing for a version history; and

providing a mountable file system for user access to full backups with version history data located on the server.

2. The method of claim 1, identifying changed files based on comparisons with the manifest.

3. The method of claim 1, identifying changed files based on meta-data retrieved from a remote server.

4. The method of claim 1, comprising compressing and encrypting the changed segment.

5. The method of claim 1, comprising transmitting the changed segment in chunks.

6. The method of claim 1, comprising compressing, encrypting and transmitted the changed segment using a plurality of parallel transmission threads.

7. The method of claim 1, comprising receiving, decrypting, and patching the changed segment in the replicated client data set.

8. The method of claim 1, comprising generating a full backup with snapshots going back in time for version history.

9. The method of claim 1, comprising generating a mountable directory or file system containing the replicated client data set.

10. The method of claim 1, comprising creating a metadata database as a shadow of a user file in a hidden folder.

11. The method of claim 10, comprising applying the meta data database for a random access item lookup.

12. The method of claim 1, comprising storing meta data in a single index database with a meta data database file size below the size of a single block.

13. The method of claim 1, comprising translating between an original native file system and a split file and metadata system.

14. The method of claim 1, comprising using a software client as a translation agent.

15. The method of claim 1, comprising storing file data and meta data separately on a file system, and linking both through folder proximity and file name.

16. The method of claim 1, comprising storing in a metadata file modification time, file creation time, file access time, file size, access control list (ACLs), permission list, hash of file data.

17. The method of claim 1, comprising storing metadata in sidecar files for any type of attribute.

18. The method of claim 1, comprising transferring back-up data files using a client software on the computer, wherein the client software receives inputs from a web user portal to control operations of the client software and a system management platform to back up the customer computer with one or more data storage silos, each including a plurality of storage directors communicating with the client software;

and a clustered data storage array.

19. The method of claim 1, comprising restoring files through a group consisting of:

the client software, a web based file browser replicating a look and feel of a native file explorer, and a remote data drive at the data storage silo mounted by a user simulating a local drive.

20. A system, comprising:

a plurality of customer computers to be backed-up, each customer computer running a client software to transfer back-up data files by identifying changed files;

translating between an original native file system and a split file and metadata system;

generating a first sub-file signature for one or more segments of a changed file;

subsequently generating a second sub-file signature for each file segment to detect insertions, removals, and rearrangements of data in the file;

transmitting one more changed segment(s) in the file;

writing a manifest file reflecting the new target state as an incremental backup;

compressing, encrypting and transmitted the changed segment through a Web-based Distributed Authoring and Versioning application program interface;

expanding patch sets, and saving the patch sets with the file system to provide an incremental backup while generating a full backup;

taking a snapshot after a full backup providing for a version history; and

providing a mountable file system for user access to full backups with version history data located on the server;

a system management platform coupled to the client software over the Internet, the system management platform receiving inputs from a web user portal to control operations of the client software and the system management platform to back up the customer computer; and

two or more data storage silos, each including;

a plurality of storage directors communicating with the client software; and

a clustered data storage array.

21. The system of claim 20, wherein the client software communicates using Web-based Distributed Authoring and Versioning to collaborative updating and management of files on remote web servers and wherein the client software selects the number of threads, and wherein the client software comprises a configurable parallelism selection for multi-processor efficiency and high latency connections, or for backing up NAS/SAN or RAID arrays with multiple disks.

22. The system of claim 20, wherein the data storage silos comprises a pair of head nodes and a plurality of storage nodes, wherein each storage node comprises a computer with disks and an on-disk encryptor to generate clear text and sent over an Internet Small Computer System Interface (iSCSI), wherein iSCSI traffic is load balanced and fails over network links and connected to a plurality of switches and wherein iSCSI data comprise whole disk mappings and wherein the head nodes create RAID stripes across the storage nodes.