Management of data replication and storage apparatuses, methods and systems

• US 9,594,636 B2
• Filed: 05/30/2014
• Issued: 03/14/2017
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Datto LLC
• Original Assignee: Datto, Inc. (Kaseya Limited)
• Inventors: Mortensen, Spencer, Giacobbi, Justin
• Primary Examiner(s): Le, Hung
• Application Number:
  US14/292,737
• Publication Number:
  US 20150347548A1
• Time in Patent Office: 1,019 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1461 Backup scheduling policy
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The MANAGEMENT OF DATA REPLICATION AND STORAGE APPARATUSES, METHODS AND SYSTEMS (“BDR”) transform machine datasets using BDR components into replications over a network distributing redundant ready-to-use backup files. In some implementations, the disclosure provides a processor-implemented method for managing the replication and storage of data sets.

Claims

1. A processor implemented method for managing replication and storage of disk images of physical or virtual storage disks, the method comprising:

sending a backup request, from a backup aggregator to a first client server at a first client site, according to a backup and snapshot schedule, wherein;

the backup aggregator has a first file system; and

the first client server has a first storage disk;

receiving, at the backup aggregator, a backup response from the first client server at the first client site, the backup response comprising a first disk image reflecting a first state and content of the first storage disk of the first client server;

capturing a first snapshot of the received first disk image, and storing the first snapshot in the first file system of the backup aggregator;

sending, from the backup aggregator to a master server of a cloud storage center, a task processing assistance request comprising a backup vector, wherein;

the backup vector comprises;

a dataset indicator corresponding to the first disk image reflecting the first state and content of the first storage disk of the first client server; and

a mirroring relationship between the first file system of the backup aggregator and a second file system of a first cloud storage server of the cloud storage center; and

the task processing assistance request is sent according to;

a first client site user predefined dataset priority order; and

a second prioritization order based on at least one type of action associated with at least one task involved in the task processing assistance request;

validating, by the master server, the backup vector received in the task processing assistance request; and

synchronizing the first file system of the backup aggregator with the second file system of the first cloud storage server only after receiving a task processing assistance response from the master server acknowledging the validating of the backup vector.

2. The method of claim 1, wherein the said received first disk image is selected from the group consisting of full backup data, differential backup data and incremental backup data.

3. The method of claim 1, wherein the captured first snapshot is associated with a time when the first disk image was generated.

4. The method of claim 3, wherein:

capturing a first snapshot comprises capturing a plurality of successive snapshots including the first snapshot; and

each snapshot of the plurality of successive snapshots serves as a restore point for the first storage disk of the first client server at a corresponding time.

5. The method of claim 4, wherein the task processing assistance request further comprises:

at least one new cryptographic key generated by the backup aggregator; and

a first plurality of metrics related to the operational state of the backup aggregator.

6. The method of claim 5, further comprising sending by the master server an assisting node processing request to the first cloud storage server, wherein the assisting node processing request comprises the at least one new cryptographic key generated by the backup aggregator.

7. The method of claim 6, further comprising receiving by the master server an assisting node processing response comprising a second plurality of metrics related to the operational state of the first cloud storage server.

8. The method of claim 7, further comprising receiving by the master server a network reconfiguration request comprising a command to delete the backup vector describing the mirroring relationship between the first file system and the second file system.

9. The method of claim 8, further comprising sending by the master server an assisting node reconfiguration request to the first cloud storage server comprising at least one command to disable at least one cryptographic key corresponding to the backup aggregator.

10. The method of claim 9, wherein a synchronization request generated by the backup aggregator is declined by the first cloud storage server.

11. The method of claim 10, further comprising sending by the master server an assisting node reconfiguration request to the backup aggregator comprising at least one command to delete a vector from a remote backup schedule.

12. The method of claim 3, further comprising generating a rapid recovery file in a ready-to-run disk image format corresponding to the first disk image received at the backup aggregator and storing the rapid recovery file in the first file system.

13. The method of claim 12, wherein:

the backup response is a first backup response;

receiving a backup response from the first client server comprises receiving, at the backup aggregator and at a first frequency, a plurality of successive backup responses including the first backup response; and

synchronizing the first file system with the second file system comprises synchronizing the first file system with the second file system at a different frequency than the first frequency.

14. The method of claim 1, wherein the captured first snapshot comprises an incremental relation mirroring data blocks that have been modified in the first storage disk from a first time when a previously received disk image was generated to a second time when the first disk image was generated, and the method further comprises associating the first snapshot with the second time when the first disk image was generated.

15. The method of claim 14, wherein:

capturing a first snapshot comprises capturing a plurality of successive snapshots including the first snapshot; and

each snapshot of the plurality of successive snapshots serves as a restore point for the first storage disk of the first client server at a corresponding time.

16. The method of claim 15, wherein the task processing assistance request further comprises:

at least one new cryptographic key generated by the backup aggregator; and

a first plurality of metrics related to the operational state of the backup aggregator.

17. The method of claim 16, further comprising sending by the master server an assisting node processing request to the first cloud storage server, wherein the assisting node processing request comprises the at least one new cryptographic key generated by the backup aggregator.

18. The method of claim 17, further comprising receiving by the master server an assisting node processing response comprising a second plurality of metrics related to the operational state of the first cloud storage server.

19. The method of claim 18, further comprising receiving by the master server a network reconfiguration request comprising a command to delete the backup vector describing the mirroring relationship between the first file system and the second file system.

20. The method of claim 19, further comprising sending by the master server an assisting node reconfiguration request to the first cloud storage server comprising at least one command to disable at least one cryptographic key corresponding to the backup aggregator.

21. The method of claim 20, wherein a synchronization request generated by the backup aggregator is declined by the first cloud storage server.

22. The method of claim 21, further comprising sending by the master server an assisting node reconfiguration request to the backup aggregator comprising at least one command to delete a vector from a remote backup schedule.

23. The method of claim 14, further comprising generating a rapid recovery file in a ready-to-run disk image format corresponding to the first disk image received at the backup aggregator and storing the rapid recovery file in the first file system.

24. The method of claim 23, wherein:

the backup response is a first backup response;

receiving a backup response from the first client server comprises receiving, at the backup aggregator and at a first frequency, a plurality of successive backup responses including the first backup response; and

synchronizing the first file system with the second file system comprises synchronizing the first file system with the second file system at a different frequency than the first frequency.

25. A processor implemented method for managing the replication and storage of data sets comprising:

sending a backup request according to a backup and snapshot schedule;

receiving at a host of a first file system a backup response comprising image enabled backup data reflecting the state and content of a computer device;

capturing a snapshot of the image enabled backup data, and storing the snapshot in the first file system;

validating according to a remote backup schedule a vector comprising a data set indicator corresponding to the computer device, and a mirroring relationship between the first file system and a second file system;

sending a task processing assistance request to a master server comprising a vector identifier, wherein the task processing assistance request is sent according to a first prioritization order comprising a user'"'"'s defined prioritized replication order and a second prioritization order comprising a prioritized actions'"'"' execution order; and

synchronizing the first file system with the second file system only after receiving a task processing assistance response from the master server validating the mirroring relationship,

wherein the captured snapshot comprises an incremental relation mirroring data blocks that have been modified in the computer system from the time when a previously received image enabled backup data was generated to the time when the current image enabled backup data was generated, and associating the snapshot with the time when the current image enabled backup data was generated,

wherein each snapshot can serve as a restore point for the computer device corresponding to its associated time,

wherein the host of the first file system is different from a host of the second file system, and

wherein the task processing assistance request further comprises at least one new cryptographic key generated by the host of the first file system and a plurality of metrics related to the operational state of the host of the first file system.

26. The method of claim 25, further comprising a step wherein said master server sends an assisting node processing request to the host of the second file system comprising at least one new cryptographic key generated by the host of the first file system.

27. The method of claim 26, further comprising a step wherein said master server receives an assisting node processing response comprising a plurality of metrics related to the operational state of the host of the second file system.

28. The method of claim 27, further comprising a step wherein said master server receives a network reconfiguration request comprising a command to delete a vector describing a relation between the first file system and the second file system.

29. The method of claim 28, further comprising a step wherein said master server sends an assisting node reconfiguration request to the host of the second file system comprising at least one command to disable at least one cryptographic key corresponding to the host of the first file system.

30. The method of claim 29, wherein a synchronization request generated by the host of the first file system is declined by the host of the second file system.

31. The method of claim 30, further comprising a step wherein said master server sends an assisting node reconfiguration request to the host of the first file system comprising at least one command to delete a vector from a remote backup schedule.

32. A processor implemented method for managing the replication and storage of data sets comprising:

sending a backup request according to a backup and snapshot schedule;

receiving at a host of a first file system a backup response comprising image enabled backup data reflecting the state and content of a computer device;

capturing a snapshot of the image enabled backup data, and storing the snapshot in the first file system;

validating according to a remote backup schedule a vector comprising a data set indicator corresponding to the computer device, and a mirroring relationship between the first file system and a second file system;

sending a task processing assistance request to a master server comprising a vector identifier, wherein the task processing assistance request is sent according to a first prioritization order comprising a user'"'"'s defined prioritized replication order and a second prioritization order comprising a prioritized actions'"'"' execution order; and

synchronizing the first file system with the second file system only after receiving a task processing assistance response from the master server validating the mirroring relationship,

wherein the image enabled backup data is stored in the first file system and the captured snapshot is associated with the time when the image enabled backup data was generated,

wherein each snapshot can serve as a restore point for the computer device corresponding to its associated time,

wherein the host of the first file system is different from a host of the second file system, and

wherein the task processing assistance request further comprises at least one new cryptographic key generated by the host of the first file system and a plurality of metrics related to the operational state of the host of the first file system.

33. The method of claim 32, further comprising a step wherein said master server sends an assisting node processing request to the host of the second file system comprising at least one new cryptographic key generated by the host of the first file system.

34. The method of claim 33, further comprising a step wherein said master server receives an assisting node processing response comprising a plurality of metrics related to the operational state of the host of the second file system.

35. The method of claim 34, further comprising a step wherein said master server receives a network reconfiguration request comprising a command to delete a vector describing a relation between the first file system and the second file system.

36. The method of claim 35, further comprising a step wherein said master server sends an assisting node reconfiguration request to the host of the second file system comprising at least one command to disable at least one cryptographic key corresponding to the host of the first file system.

37. The method of claim 36, wherein a synchronization request generated by the host of the first file system is declined by the host of the second file system.

38. The method of claim 37, further comprising a step wherein said master server sends an assisting node reconfiguration request to the host of the first file system comprising at least one command to delete a vector from a remote backup schedule.

39. In a system comprising a master server, a storage server communicatively coupled to the master server via a network, and a backup aggregator communicatively coupled to the master server and the storage server via the network, wherein the backup aggregator has a snapshot-capable file system, a method for replicating on the backup aggregator a plurality of content states of a client computing device and managing storage, on the storage server, of at least some of the replicated plurality of content states of the client computing device, wherein the client computing device is communicatively coupled to the backup aggregator via the network, the method comprising:

A) receiving at the backup aggregator, from the client computing device via the network, a succession of image-based backup files respectively representing the plurality of content states of the client computing device as a function of time;

B) taking respective snapshots, at the backup aggregator, of the succession of image-based backup files, and storing the respective snapshots in the snapshot-capable file system of the backup aggregator;

C) transmitting from the backup aggregator, to the master server via the network, a task processing assistance request including a mirroring relationship stored at the backup aggregator, wherein the mirroring relationship specifies an identifier for the backup aggregator and at least one destination identifier for the storage server;

D) in response to C), receiving at the backup aggregator, from the master server via the network, a task processing assistance response including either an acknowledgement of the mirroring relationship or an instruction to remove the mirroring relationship stored at the backup aggregator; and

E) transmitting from the backup aggregator, to the storage server via the network, a send file representing differential changes between the at least some of the respective snapshots of the succession of image-based backup files if the task processing assistance response received in D) includes the acknowledgement of the mirroring relationship,

wherein in C), the task processing assistance request is transmitted by the backup aggregator to the master server based at least in part on a client-defined task prioritization order relating to a criticality of the plurality of content states of the client computing device.

40. A backup aggregator to replicate a plurality of content states of a client computing device and to manage storage, on a storage server, of at least some of the replicated plurality of content states of the client computing device, the backup aggregator comprising:

at least one communication interface;

at least one memory to store;

processor-executable instructions;

a mirroring relationship that specifies an identifier for the backup aggregator and at least one destination identifier for the storage server; and

a client-defined task prioritization order relating to a criticality of the plurality of content states of the client computing device; and

at least one processor communicatively coupled to the at least one communication interface and the at least one memory, wherein upon execution by the at least one processor of the processor-executable instructions, the at least one processor controls the at least one memory so as to implement a snapshot-capable file system for the backup aggregator, and wherein the at least one processor further;

A) controls the at least one communication device to receive from the client computing device a succession of image-based backup files respectively representing the plurality of content states of the client computing device as a function of time;

B) takes respective snapshots of the succession of image-based backup files;

C) controls the at least one memory to store the respective snapshots in the snapshot-capable file system;

D) controls the at least one communication interface to transmit to a master server a task processing assistance request including the mirroring relationship;

E) controls the at least one communication interface to receive from the master server a task processing assistance response including either an acknowledgement of the mirroring relationship or an instruction to remove the mirroring relationship stored in the at least one memory; and

F) controls the at least one communication interface to transmit to the storage server a send file representing differential changes between the at least some of the respective snapshots of the succession of image-based backup files if the task processing assistance response received in E) includes the acknowledgement of the mirroring relationship,

wherein in D), the at least one processor controls the at least one communication interface to transmit the task processing assistance request to the master server based at least in part on the client-defined task prioritization order stored in the at least one memory and relating to the criticality of the plurality of content states of the client computing device.