Optimizing data transmission bandwidth consumption over a wide area network

• US 8,468,135 B2
• Filed: 04/14/2010
• Issued: 06/18/2013
• Est. Priority Date: 04/14/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Constantinescu, Mihail Corneliu, Glider, Joseph Samuel
• Primary Examiner(s): Hwa, Shyue Jiunn
• Assistant Examiner(s): Gurmu, Muluemebet
• Application Number:
  US12/760,489
• Publication Number:
  US 20110258161A1
• Time in Patent Office: 1,161 Days
• Field of Search: 709/236, 707/640
• US Class Current: 707/640
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: H04L 41/0823 characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability for optimising operational conditions of wireless networks H04W24/02
• CPC Class Codes: H04L 47/12 Avoiding congestion; Recovering from congestion
• CPC Class Codes: H04L 47/34 ensuring sequence integrity, e.g. using sequence numbers

Abstract

An exemplary embodiment includes partitioning a data message to be communicated from a first data site to a second data site into data chunks; generating a data chunk identifier for each data chunk; determining whether the data chunks are stored at the second data site; when at least one data chunk is not stored at the second data site, adding the data chunk identifier for each data chunk not stored at the second data site to a data structure at the first data site; sending a transformed data message from the first date site to the second data site; wherein, when at least one data chunk is already stored at the second data site, rather than including that data chunk, the transformed data message instead includes at least one tuple to enable the data message to be reconstructed at the second data site without sending the previously stored data chunk, the transformed data message also includes each data chunk not stored at the second data site.

Claims

1. A method of reducing bandwidth consumption for data being sent between a client and server over a wide area network (WAN) by optimizing deduplication performance, the method comprising:

partitioning a data message into a plurality of data chunks, said message to be sent from a client at a first data site to a server at a second data site over a WAN;

generating a data chunk identifier (ID) for each of said chunks;

determining whether said chunks are stored at the second data site;

when at least one of said chunks is not stored at the second data site, adding said ID for each of said chunks not stored at the second data site to a data structure at the first data site;

when at least two data chunks partitioned from said message are in sequence and are not stored at the second data site, linking in the data structure at the first data site, the data chunk ID for each data chunk in sequence that are not stored at the second data site in an order corresponding to the sequence in said message; and

sending a transformed data message from the first date site to the second data site, wherein said transformed message includes at least one tuple when at least one of said chunks is stored at the second data site and data chunks among said chunks that are not stored at the second data site, wherein said tuple;

is a paired representation of data chunk sequence, is to be used to reconstruct said message, and includes a first data chunk ID and a sequence count that represents an aggregate number of linked data chunk IDs.

2. The method of claim 1, further comprising:

when said transformed message comprises at least two tuples, ordering the at least two tuples (in said transformed message) in an order corresponding to an order said chunks were partitioned from said message.

3. The method of claim 1, further comprising:

when said transformed message includes at least two data chunks, ordering the at least two data chunks (in said transformed message) in an order corresponding to an order said chunks were partitioned from said message.

4. The method of claim 1, further comprising:

when said transformed message includes at least one data chunk and at least one tuple, ordering the at least one tuple and the at least one data chunk (in said transformed message) in an order corresponding to an order said chunks were partitioned from said message.

5. The method of claim 1, wherein said determining includes searching for the data chunk ID generated for each of said chunks in the data structure at the first data site.

6. The method of claim 1, wherein said sending is performed as part of an operation to remotely host computing services at the second data site for the first data site.

7. The method of claim 1, wherein said sending backs up data from the first data site to the second data site.

22. The method of claim 1, wherein said sending thereby optimizes deduplication performance such that the number of data chunk IDs sent and accesses needed to the data structure at the second data site, for reconstructing said message at the second data site, are minimized.

8. A method of reducing bandwidth consumption for data being sent between a client and server over a wide area network (WAN) by optimizing deduplication performance, the method comprising:

receiving a transformed data message at a second data site, said transformed message;

being previously sent from a client at a first data site to a server at the second data site over a WAN, created from data message partitioned into a plurality of data chunks, and includes data chunks from the data message not stored at the second data site and at least one tuple when at least one data chunk from the data message is stored at the second data site, wherein said tuple;

is a paired representation of data chunk sequence, is to be used at the second data site to reconstruct said transformed, and includes a first data chunk ID and a sequence count that represents an aggregate number of linked data chunk IDs; and

when said transformed message includes at least one data chunk;

generating a data chunk identifier (ID) for each chunk in said transformed message, adding said ID for each chunk in said transformed message to a data structure at the second data site, and storing each chunk in said transformed message at the second data site; and

reconstructing said transformed message at the second data site by;

when said transformed message includes at least one data chunk, assembling said at least one chunk into a reconstructed data message in an order corresponding to an order in said transformed message,

when said transformed message includes at least one tuple, assembling each chunk corresponding to a chunk ID referenced in the at least one tuple into said reconstructed message in an order the at least one tuple is in said transformed message.

9. The method of claim 8, further comprising:

performing a data restore from the second data site to a first data site using at least one data chunk from said transformed message stored at the second data site.

10. The method of claim 8, wherein said receiving is performed as part of an operation to remotely host computing services at the second data site for the first data site.

11. A system of reducing bandwidth consumption for data being sent between a client and server over a wide area network (WAN) by optimizing deduplication performance, the system comprising:

a storage repository at a first data site storing data chunks and a data structure includes data chunk identifiers of the stored data chunks; and

a data deduplication node at the first data site that;

(i) partitions a data message, to be sent from a client at the first data site to a sever at a second data site over a WAN, into a plurality of data chunks, (ii) generates a data chunk identifier (ID) for each of said chunks, determines whether said chunks are stored at the second data site, (iii) adds an ID for each data chunk not stored at the second data site to the data structure at the first data site, when at least one of said chunks that are not stored at the second data site, (iv) links in the data structure at the first data site, the data chunk ID for each data chunk in sequence that are not stored at the second data site, in an order corresponding to the sequence in said message when at least two data chunks partitioned from said message are in sequence and are not stored at the second data site, and (v) sends a transformed data message from the first date site to the second data site, wherein said transformed message includes at least one tuple when at least one of said chunks is stored at the second data site, and data chunks among said chunks that are not stored at the second data site, wherein said tuple;

is a paired representation of data chunk sequence, is to be used to reconstruct said message, and includes a first data chunk ID and a sequence count that represents an aggregate number of linked data chunk IDs.

12. The system of claim 11, further comprising:

a storage repository at the second data site storing a data structure that includes data chunk identifiers and data chunks in said transformed message; and

a data deduplication node at the second data site, in communication with the data deduplication node at the first site;

receives said transformed message, generates a data chunk identifier (ID) for each data chunk in the transformed data message, when said transformed message includes at least one data chunk, and adds the ID for each data chunk in said transformed message to the data structure at the second data site.

13. The system of claim 11, further comprising:

at least one client computing node, in communication with the deduplication node at the first data site, that;

sends a data message to be sent from the first data site to the second data site, and receives a data message to be sent from the second data site to the first data site.

14. The system of claim 11, further comprising:

a server computing node, in communication with the deduplication node at the second data site, that sends a data message to be restored from the second data site to the first data site using at least one data chunk in the transformed message stored at the second data site.

15. The system of claim 11, wherein said storage repository at the first data site stores data chunks to be restored from second data site to the first data site.

16. The system of claim 11, wherein said determines includes searching for the data chunk ID generated for each of said chunks in the data structure at the first data site.

17. The system of claim 12, wherein the deduplication node at the second data site reconstructs said transformed message, said includes:

when said transformed message includes at least one data chunk, assembling said at least one chunk into a reconstructed data message in an order corresponding to an order in said transformed message, and

when said transformed message includes at least one tuple, assembling each chunk corresponding to a chunk ID referenced in the at least one tuple into said reconstructed message in an order the at least one tuple is in said transformed message.

18. The system claim 12, wherein the second data site remotely hosts computing services for the first data site.

19. A computer program product of reducing bandwidth consumption for data being sent between a client and server over a wide area network (WAN) by optimizing deduplication performance, the computer program product comprising a computer readable storage medium having computer readable program code embodied therewith, wherein said program code upon being processed on a computer causes the computer to:

partition a data message into a plurality of data chunks, said message to be sent from a client at a first data site to a server at a second data site over a WAN;

generate a data chunk identifier (ID) for each of said chunks;

determine whether said chunks are stored at the second data site;

add said ID for each of said chunks not stored at the second data site to a data structure at the first data site, when at least one of said chunks is not stored at the second data site; and

link in the data structure at the first data site, the data chunk ID for each data chunk in sequence that are not stored at the second data site, in an order corresponding to the sequence in said message when at least two data chunks partitioned from said message are in sequence and are not stored at the second data site

send a transformed data message from the first date site to the second data site, wherein said transformed message includes at least one tuple when at least one of said chunks is stored at the second data site and data chunks among said chunks that are not stored at the second data site, wherein said tuple;

is a paired representation of data chunk sequence, is to be used to reconstruct said message, and includes a first data chunk ID and a sequence count that represents an aggregate number of linked data chunk IDs.

20. The computer program product of claim 19, wherein said program code further causes the computer to:

receive said transformed message at the second data site;

generate an ID for each of said chunks in said transformed message and add each said ID to a data structure at the second data site when said transformed message includes at least one data chunk; and

store each data chunk in the transformed message.

21. The computer program product of claim 19, wherein said program code further causes the computer to:

reconstruct said transformed message at the second data site, wherein said reconstructing includes;

when said transformed message includes at least one data chunk, assembling said at least one chunk into a reconstructed data message in an order corresponding to an order in said transformed message, and

when said transformed message includes at least one tuple, assembling each chunk corresponding to a chunk ID referenced in the at least one tuple into said a reconstructed message in an order the at least one tuple is in said transformed message.