Optimizing Data Transmission Bandwidth Consumption Over a Wide Area Network

• US 20110258161A1
• Filed: 04/14/2010
• Published: 10/20/2011
• Est. Priority Date: 04/14/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Constantinescu, Mihail Corneliu, Glider, Joseph Samuel
• Granted Patent: US 8,468,135 B2
• Time in Patent Office: Days
• US Class Current: 707/640
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: H04L 41/0823 characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability for optimising operational conditions of wireless networks H04W24/02
• CPC Class Codes: H04L 47/12 Avoiding congestion; Recovering from congestion
• CPC Class Codes: H04L 47/34 ensuring sequence integrity, e.g. using sequence numbers

Abstract

An exemplary embodiment includes partitioning a data message to be communicated from a first data site to a second data site into data chunks; generating a data chunk identifier for each data chunk; determining whether the data chunks are stored at the second data site; when at least one data chunk is not stored at the second data site, adding the data chunk identifier for each data chunk not stored at the second data site to a data structure at the first data site; sending a transformed data message from the first date site to the second data site; wherein, when at least one data chunk is already stored at the second data site, rather than including that data chunk, the transformed data message instead includes at least one tuple to enable the data message to be reconstructed at the second data site without sending the previously stored data chunk, the transformed data message also includes each data chunk not stored at the second data site.

Claims

1. A method for optimizing data transmission bandwidth consumption over a wide area network, comprising:

partitioning a data message to be communicated from a first data site to a second data site into a plurality of data chunks;

generating a data chunk identifier for each of the plurality of data chunks;

determining whether the plurality of data chunks are stored at the second data site;

when at least one data chunk is not stored at the second data site, adding the data chunk identifier for each data chunk not stored at the second data site to a data structure at the first data site; and

sending a transformed data message from the first date site to the second data site, wherein the transformed data message comprises;

when at least one of the plurality of data chunk is stored at the second data site, at least one tuple, wherein the at least one tuple is to be used to reconstruct the data message, and

when at least one data chunk is not stored at the second data site, the at least one data chunk not stored at the second site.

2. The method of claim 1, further comprising:

when at least two data chunks partitioned from the data message are in sequence and are not stored at the second data site, linking in the data structure at the first data site the data chunk identifier for each data chunk in sequence and not stored at the second data site in an order corresponding to the sequence.

3. The method of claim 1, wherein the tuple comprises a first data chunk identifier and a sequence count, wherein the sequence count represents an aggregate number of linked data chunk identifiers.

4. The method of claim 1, further comprising:

when the transformed data message comprises at least two tuples, ordering the tuples in the transformed data message in an order corresponding to an order the plurality of data chunks were partitioned in the data message.

5. The method of claim 1, further comprising:

when the transformed data message comprises at least two data chunks, ordering the data chunks in the transformed data message in an order corresponding to an order the plurality of data chunks were partitioned in the data message.

6. The method of claim 1, further comprising:

when the transformed data message comprises at least one data chunk and at least one tuple, ordering the at least one tuple and the at least one data chunk in the transformed data message in an order corresponding to an order the plurality of data chunks were partitioned in the data message.

7. The method of claim 1, wherein determining whether the plurality of data chunks are stored at the second data site comprises:

searching for the data chunk identifier generated for each of the plurality of data chunks in the data structure at the first data site.

8. The method of claim 1, wherein sending the transformed data message is sent from the first date site to the second data site is performed as part of an operation to remotely host computing services at the second data site for the first data site.

9. The method of claim 1, wherein sending the transformed data message from the first date site to the second data site is performed as part of an operation to backup data from the first data site to the second data site.

10. A method for optimizing data transmission bandwidth consumption over a network, comprising:

receiving a transformed data message at a second data site; and

when the transformed data message comprises at least one data chunk,

generating a data chunk identifier for each data chunk in the transformed data message,

adding the data chunk identifier for each data chunk in the transformed data message to a data structure at the second data site, and

storing each data chunk in the transformed message in a storage repository at the second data site.

11. The method of claim 10, further comprising:

reconstructing the transformed data message at the second data site, wherein reconstructing the transformed data message comprises;

when the transformed data message comprises at least one data chunk, assembling the at least one data chunk in the transformed data message into a reconstructed data message in an order the at least one data chunk appear in the transformed data message,

when the transformed data message comprises at least one tuple, assembling at least one data chunk corresponding to at least one data chunk identifier in the at least one tuple in the transformed data message into a reconstructed data message in an order the at least one tuple appears in the transformed data message.

12. The method of claim 10, further comprising:

performing a data restore from the second data site to a first data site using at least one data chunk in the transformed message stored at the second data site.

13. The method of claim 11, wherein receiving the transformed data message at the second data site is performed as part of an operation to remotely host computing services at the second data site for a first data site.

14. A system for optimizing data transmission bandwidth consumption over a wide area network, comprising:

a data structure at a first data site configured to store a plurality of data chunk identifiers; and

a data deduplication node at the first data site, the deduplication node comprises;

a data partition module configured to partition a data message to be communicated from the first data site to a second data site into a plurality of data chunks,

a data chunk identifier generation module coupled to the data partition module and configured to generate a data chunk identifier for each of the plurality of data chunks,

a determination module coupled to the data chunk identifier generation module and configured to determine whether the plurality of data chunks are stored at the second data site,

a data structure management module coupled to the determination module and configured to add the data chunk identifier for each data chunk not stored at the second data site to the data structure at the first data site when at least one of the plurality of data chunk is not stored at the second data site, and

a transmission module coupled to the data structure management module and configured to send a transformed data message from the first date site to the second data site, wherein the transformed data message comprises;

when at least one data chunk is stored at the second data site, at least one tuple, wherein the at least one tuple is to be used to reconstruct the data message, and

when at least one data chunk is not stored at the second data site, the at least one data chunk not stored at the second site.

15. The system of claim 14, further comprising:

a data structure at the second data site and configured to store a plurality of data chunk identifiers;

a data deduplication node at the second data site coupled to the deduplication node at the first data site over a network, the deduplication node at the second site comprises;

a transmission module configured to receive the transformed data message at the second data site,

a data chunk identifier generation module coupled to the receiving module and configured to generate a data chunk identifier for each data chunk in the transformed data message, when the transformed data message comprises at least one data chunk, and

a data structure management module coupled to the data chunk identifier generation module and configured to add the data chunk identifier for each data chunk in the transformed data message to the data structure at the second data site; and

a storage repository coupled to deduplication node at the second data site and configured to store each data chunk in the transformed data message.

16. The system of claim 14, further comprising:

at least one client computing node coupled to the deduplication node at the first data site, the at least one client computing node configured to;

send a data message to be communicated from the first data site to the second data site, and

receive a data message to be communicated from the second data site to the first data site.

17. The system of claim 14, further comprising:

a backup computing node coupled to the deduplication node at the second data site and configured to send a data message to be restored from the second data site to the first data site using at least one data chunk in the transformed message stored at the second data site.

18. The system of claim 14, the system further comprising:

a storage repository coupled to the deduplication node at the first data site and configured to store a plurality of data chunks to be restored from second data site to the first data site.

19. The system of claim 14, wherein determining whether the plurality of data chunks are stored at the second data site comprises:

searching for the data chunk identifier generated for each of the plurality of data chunks in the data structure at the first data site.

20. The system of claim 15, wherein the deduplication node at the second data site further comprises:

a data reconstruction module coupled to the data structure management module at the second data site and configured to reconstruct the transformed data message, wherein reconstructing the transformed data message comprises;

when the transformed data message comprises at least one data chunk, assembling the at least one data chunk in the transformed data message into a reconstructed data message in an order the at least one data chunk appears in the transformed data message,

when the transformed data message comprises at least one tuple, assembling at least one data chunk corresponding to at least one data chunk identifier in the at least one tuple in the transformed data message into a reconstructed data message in an order the at least one tuple appears in the transformed data message.

21. The system claim 15, wherein the second data site remotely hosts computing services for the first data site.

22. A computer program product for optimizing data transmission bandwidth consumption over a wide area network, comprising:

a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code comprising;

computer readable program code configured to partition a data message to be communicated from a first data site to a second data site into a plurality of data chunks,

computer readable program code configured to generate a data chunk identifier for each of the plurality of data chunks,

computer readable program code configured to determine whether the plurality of data chunks are stored at the second data site,

computer readable program code configured to add the data chunk identifier for each data chunk not stored at the second data site to a data structure at the first data site, when at least one data chunk is not stored at the second data site; and

computer readable program code configured to send a transformed data message from the first date site to the second data site, wherein the transformed data message comprises;

when at least one data chunk is stored is stored at the second data site, at least one tuple, wherein the at least one tuple is to be used to reconstruct the data message, and

when at least one data chunk is not stored at the second data site, the at least one data chunk not stored at the second site.

23. The computer program product of claim 22, further comprising:

computer readable program code configured to receive the transformed data message at the second data site;

computer readable program code configured to generate a data chunk identifier for each data chunk in the transformed data message and add the data chunk identifier for each data chunk in the transformed data message to a data structure at the second data site, when the transformed data message comprises at least one data chunk; and

computer readable program code configured to store each data chunk in the transformed message.

24. The computer program product of claim 22, further comprising:

computer readable program code configured to reconstruct the transformed data message at the second data site, wherein reconstructing the transformed data message comprises;

when the transformed data message comprises at least one data chunk, assembling the at least one data chunk in the transformed data message into a reconstructed data message in an order the at least one data chunk appear in the transformed data message,

when the transformed data message comprises at least one tuple, assembling at least one data chunk corresponding to at least one data chunk identifier in the at least one tuple in the transformed data message into a reconstructed data message in an order the at least one tuple appears in the transformed data message.