Cloud based data migration and replication

• US 9,075,529 B2
• Filed: 01/04/2013
• Issued: 07/07/2015
• Est. Priority Date: 01/04/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Karve, Shrikant V., Kulkarni, Janmejay S., Patel, Sarvesh S., Pathak, Ashish R., Patil, Sandeep R.
• Primary Examiner(s): Pollack, Melvin H
• Assistant Examiner(s): GIDADO, OLUWATOSIN M
• Application Number:
  US13/733,953
• Publication Number:
  US 20140195636A1
• Time in Patent Office: 914 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/108 characterised by resources being split in blocks or fragments
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

A mechanism is provided in a data processing system for data replication. The mechanism divides a set of data to be replicated into a plurality of slices. Each slice of the plurality of slices is to be copied from a primary site to a master secondary site within a plurality of secondary sites. The plurality of secondary sites comprises the master secondary site and at least one subsidiary peer site. The mechanism copies the plurality of slices from the primary site to the plurality of secondary sites such that the master secondary site stores a first slice of the set of data and each of the at least one subsidiary peer site stores a corresponding slice of the set of data. Responsive to completion of copying the plurality of slices to the plurality of secondary sites, for each of the at least one subsidiary peer site, the mechanism initiates copying of the corresponding slice of the set of data to the master secondary site.

Claims

1. A method, in a data processing system, for data replication, comprising:

dividing a set of data to be replicated into a plurality of slices, wherein each slice of the plurality of slices is to be copied from a primary site to a master secondary site within a plurality of secondary sites, wherein the plurality of secondary sites comprises the master secondary site and at least one subsidiary peer site;

copying the plurality of slices from the primary site to the plurality of secondary sites such that the master secondary site stores a first slice of the set of data and each of the at least one subsidiary peer site stores a corresponding slice of the set of data; and

responsive to completion of copying the plurality of slices to the plurality of secondary sites, for each given subsidiary peer site within the at leas one subsidiary peer site, initiating copying of the corresponding slice of the set of data from the given subsidiary peer site to the master secondary site.

2. The method of claim 1, wherein the primary site comprises a storage cloud.

3. The method of claim 1, wherein the master secondary site comprises a storage cloud.

4. The method of claim 1, wherein the at least one subsidiary peer site comprises a storage cloud.

5. The method of claim 1, wherein dividing the set of data into the plurality of slices comprises:

determining a number of secondary sites;

determining a storage space per secondary site based on the number of secondary sites; and

for each given secondary site, responsive to available storage space of the given secondary site being greater than or equal to the storage space, marking a region of storage space of the given secondary site for replication storage.

6. The method of claim 5, wherein dividing the set of data into the plurality of slices further comprises:

responsive to available storage space of the given secondary site not being greater than or equal to the storage space, determining a space difference; and

responsive to a next secondary site having additional storage space, shifting a next slice to start on a previous slice end and setting a new slice length for the given secondary site and the next secondary site.

7. The method of claim 1, further comprising:

responsive to completion of copying the plurality of slices to the plurality of secondary sites, enabling logic for a consolidated read view of the set of data at the master secondary site, wherein the logic presents a consolidated read view of the set of data to a client as if the set of data is stored at the master secondary site.

8. The method of claim 7, further comprising:

responsive to completion of copying slices of the set of data from the at least one subsidiary peer site to the master secondary site, disabling the logic for the consolidated read view of the set of data at the master secondary site.

9. The method of claim 1, further comprising:

establishing a partnership between the primary site and each of the plurality of secondary sites.

10. The method of claim 1, further comprising:

establishing a partnership between each of the at least one subsidiary peer site and the master secondary site.

11. The method of claim 1, wherein initiating copying of the corresponding slice of the set of data to the master secondary site comprises initiating a background copy thread on the given subsidiary peer site.

12. The method of claim 1, further comprising:

verifying data from each of the at least one subsidiary peer site is copied to the master secondary site.

13. A computer program product comprising a non-transitory computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

divide a set of data to be replicated into a plurality of slices, wherein each slice of the plurality of slices is to be copied from a primary site to a master secondary site within a plurality of secondary sites, wherein the plurality of secondary sites comprises the master secondary site and at least one subsidiary peer site;

copy the plurality of slices from the primary site to the plurality of secondary sites such that the master secondary site stores a first slice of the set of data and each of the at least one subsidiary peer site stores a corresponding slice of the set of data; and

responsive to completion of copying the plurality of slices to the plurality of secondary sites, for each given subsidiary peer site within the at least one subsidiary peer site, initiate copying of the corresponding slice of the set of data from the given subsidiary peer site to the master secondary site.

14. The computer program product of claim 13, wherein at least one of the primary site or the master secondary site comprises a storage cloud.

15. The computer program product of claim 13, wherein the at subsidiary peer site comprises a storage cloud.

16. The computer program product of claim 13, wherein dividing the set of data into the plurality of slices comprises:

determining a number of secondary sites;

determining a storage space per secondary site based on the number of secondary sites; and

for each given secondary site, responsive to available storage space of the given secondary site being greater than or equal to the storage space, marking a region of storage space of the given secondary site for replication storage.

17. The computer program product of claim 16, wherein dividing the set of data into the plurality of slices further comprises:

responsive to available storage space of the given secondary site not being greater than or equal to the storage space, determining a space difference; and

responsive to a next secondary site having additional storage space, shifting a next slice to start on a previous slice end and setting a new slice length for the given secondary site and the next secondary site.

18. The computer program product of claim 13, wherein the computer readable program further causes the computing device to:

responsive to completion of copying the plurality of slices to the plurality of secondary sites, enable logic for a consolidated read view of the set of data at the master secondary site, wherein the logic presents a consolidated read view of the set of data to a client as if the set of data is stored at the master secondary site; and

responsive to completion of copying slices of the set of data from the at least one subsidiary peer site to the master secondary site, disable the logic for the consolidated read view of the set of data at the master secondary site.

19. The computer program product of claim 13, Wherein the computer readable program further causes the computing device to:

establish a partnership between the primary site and each of the plurality of secondary sites; and

establish a partnership between each of the at least one subsidiary peer site and the master secondary site.

20. The computer program product of claim 13, wherein the computer readable program is stored in a computer readable storage medium in a data processing system and wherein the computer readable program was downloaded over a network from a remote data processing system.

21. The computer program product of claim 13, wherein the computer readable program is stored in a computer readable storage medium in a server data processing system and wherein the computer readable program is downloaded over a network to a remote data processing system for use in a computer readable storage medium with the remote system.

22. An apparatus, comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

divide a set of data to be replicated into a plurality of slices, wherein each slice of the plurality of slices is to be copied from a primary site to a master secondary site within a plurality of secondary sites, wherein the plurality of secondary sites comprises the master secondary site and at least one subsidiary peer site;

copy the plurality of slices from the primary site to the plurality of secondary sites such that the master secondary site stores a first slice of the set of data and each of the at least one subsidiary peer site stores a corresponding slice of the set of data; and

responsive to completion of copying the plurality of slices to the plurality of secondary sites, for each given subsidiary peer site within the at least one subsidiary peer site, initiate copying of the corresponding slice of the set of data from the given subsidiary peer site to the master secondary site.

23. The apparatus of claim 22, wherein at least one of the primary site or the master secondary site comprises a storage cloud.

24. The apparatus of claim 22, wherein the at least one subsidiary peer site comprises a storage cloud.

25. The apparatus of claim 22, wherein dividing the set of data into the plurality of slices comprises:

determining a number of secondary sites;

determining a storage space per secondary site based on the number of sec sites; and

for each given secondary site, responsive to available storage space of the given secondary site being greater than or equal to the storage space, marking a region of storage space of the given secondary site for replication storage.

26. The apparatus of claim 25, wherein dividing the set of data into the plurality of slices further comprises:

responsive to available storage space of the given secondary site not being greater than or equal to the storage space, determining a space difference; and

responsive to a next secondary site having additional storage space, shifting a next slice to start on a previous slice end and setting a new slice length for the given secondary site and the next secondary site.

27. The apparatus of claim 22, wherein the instructions further cause processor to:

responsive to completion of copying the plurality of slices to the plurality of secondary sites, enable logic for a consolidated read view of the set of data at the master secondary site, wherein the logic presents a consolidated read view of the set of data to a client as if the set of data is stored at the master secondary site; and

responsive to completion of copying slices of the set of data from the at least one subsidiary peer site to the master secondary site, disable the logic for the consolidated read view of the set of data at the master secondary site.

28. The apparatus of claim 22, wherein the instructions further cause the processor to:

establish a partnership between the primary site and each of the plurality of secondary sites; and

establish a partnership between each of the at least one subsidiary peer site and the master secondary site.