Reliable and Scalable Image Transfer For Data Centers With Low Connectivity Using Redundancy Detection

• US 20140222953A1
• Filed: 02/06/2013
• Published: 08/07/2014
• Est. Priority Date: 02/06/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Karve, Alexei A., Kochut, Andrzej, Shaikh, Hidayatullah, Hrischuk, Curtis E.
• Granted Patent: US 9,274,818 B2
• Time in Patent Office: Days
• US Class Current: 709/217
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 2009/45595 Network integration; Enabling network access in virtual machine instances
• CPC Class Codes: G06F 9/455 Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G06F 9/461 Saving or restoring of program or task context
• CPC Class Codes: G06F 9/4856 resumption being on a different machine, e.g. task migration, virtual machine migration G06F9/5088 takes precedence

Abstract

A system and method for efficiently transferring virtual machine images across nodes in a cloud computing environment, includes analyzing each image on each node to create hash code clusters and a similarity matrix. An instruction to transfer an image from a source node to a target node is received. The clusters and the similarity matrix are used to determine to what extent the data from the image is already on the source node, or on any other node, and further determines the cost and speed of transferring such data to the target node. An optimal transfer plan is generated, and data that is not already on the target node is transferred to the target node from the most efficient node on which it is available, according to the optimal transfer plan.

Claims

1. A method for transferring virtual machine images between a plurality of nodes, comprising:

analyzing each of a plurality of virtual machine images present on a plurality of nodes to create image clusters corresponding to a plurality of data blocks present in the plurality of virtual machine images, using a program executable by a processor communicating with a computer, the plurality of nodes including a source node and a target node;

generating a similarity matrix using the created image clusters;

selecting a virtual machine image present on the source node to be reconstituted on the target node;

identifying a plurality of data block sets in the selected virtual machine image for transfer to the target node, using the similarity matrix;

transferring the plurality of data block sets to the target node, according to an optimal transfer plan; and

reconstituting the plurality of data block sets on the target node using at least the transferred plurality of data block sets.

2. The method of claim 1, wherein the data block sets are distributed on a plurality of nodes.

3. The method of claim 1, wherein the data block sets are on both the source node and the target node.

4. The method of claim 1, wherein the data block sets are only on the source node.

5. The method of claim 1, further comprising:

identifying a first data block set from the selected virtual machine image that are present only on the source node;

identifying a second data block set from the selected virtual machine image that are present on the source node, and on at least another node excluding the target node;

identifying a third data block set from the selected virtual machine image that are present on the source node, and on at least the target node;

transferring the first and second data block sets from the source node or at least another node, excluding the target node, to the target node, according to an optimal plan; and

reconstituting on the target node the selected virtual machine image, using the transferred first and second data block sets, and the third data block set present on the target node.

6. The method of claim 1, wherein the plurality of nodes are datacenters within a public or private cloud computing environment.

7. The method of claim 1, wherein the number of image clusters used to generate a similarity matrix is configurable.

8. The method of claim 1, wherein the optimal transfer plan is determined by minimizing a time factor, a bandwidth factor, and a cost factor, using the similarity matrix, according to a defined transfer policy.

9. The method of claim 8, wherein the defined transfer policy is set by a user.

10. The method of claim 1, wherein the plurality of nodes comprises a group of systems categorized by cost and/or bandwidth requirements.

11. The method of claim 1, wherein each image cluster is comprised of a series of 20-byte SHA1 hash codes corresponding to 512-byte to 8-kilobyte sized data blocks on the plurality of virtual machine images.

12. A system for transferring virtual machine images, comprising:

a computer having a processor, and a computer-readable storage device;

a program embodied on the storage device for execution by the processor, the program having a plurality of program modules, including;

an analyzing module configured to analyze each of a plurality of virtual machine images present on a plurality of nodes to create image clusters corresponding to a plurality of data blocks present in the plurality of virtual machine images, the plurality of nodes including a source node and a target node;

a generating module configured to generate a similarity matrix using the created image clusters;

a selecting module configured to select a virtual machine image present on the source node to be reconstituted on the target node;

an identifying module configured to identify a plurality of data block sets in the selected virtual machine image for transfer to the target node, using the similarity matrix;

a transferring module configured to transfer the plurality of data block sets to the target node, according to an optimal transfer plan; and

a reconstituting module configured to reconstitute the plurality of data block sets on the target node, using at least the transferred plurality of data block sets.

13. The system of claim 12, wherein the data block sets are distributed on a plurality of nodes.

14. The system of claim 12, wherein the data block sets are on both the source node and the target node.

15. The system of claim 12, wherein the data block sets are only on the source node.

16. The system of claim 12, wherein the plurality of nodes are datacenters within a public or private cloud computing environment.

17. The system of claim 12, wherein the plurality of nodes comprises a group of systems categorized by cost and/or bandwidth requirements.

18. A computer program product for transferring virtual machine images, the computer program product comprising a computer-readable storage medium having program code embodied therewith, the program code readable/executable by a processor to perform a method comprising:

analyzing each of a plurality of virtual machine images present on a plurality of nodes, by the processor, to create image clusters corresponding to a plurality of data blocks present in the plurality of virtual machine images, the plurality of nodes including a source node and a target node;

generating a similarity matrix, by the processor, using the created image clusters;

selecting a virtual machine image present on the source node, by the processor, to be reconstituted on the target node;

identifying a plurality of data block sets in the selected virtual machine image, by the processor, for transfer to the target node, using the similarity matrix;

transferring the plurality of data block sets to the target node, by the processor, according to an optimal transfer plan; and

reconstituting the plurality of data block sets, by the processor, on the target node using at least the transferred plurality of data block sets.

19. The computer program product of claim 18, wherein the data blocks sets are distributed on a plurality of nodes.

20. The computer program product of claim 18, wherein the data block sets are on both the source node and the target node.

21. The computer program product of claim 18, wherein the data block sets are only on the source node.

22. The computer program product of claim 18, wherein the plurality of nodes are datacenters within a public or private cloud computing environment.

23. The computer program product of claim 18, wherein the number of image clusters used to generate a similarity matrix is configurable.

24. The computer program product of claim 18, the optimal transfer plan is determined by minimizing a time factor, a bandwidth factor, and a cost factor, using the similarity matrix, according to a defined transfer policy.

25. The computer program product of claim 24, wherein the defined transfer policy is set by a user.