DE-DUPLICATION DATA BANK

• US 20140222770A1
• Filed: 04/05/2014
• Published: 08/07/2014
• Est. Priority Date: 04/05/2013
• Status: Active Grant

• Current Assignee: Catalogic Software, Inc.
• Original Assignee: Syncsort Incorporated
• Inventors: Liu, Peter Chi-Hsiung, Lad, Kamlesh
• Granted Patent: US 10,366,072 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: E21D 20/02 with provisions for grouting
• CPC Class Codes: E21D 21/008 Anchoring or tensioning means
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity

Abstract

Facility for transferring data over a network between two network endpoints by transferring hash signatures over the network instead the actual data. The hash signatures are pre-generated from local static data and stored in a hash database before any data is transferred between source and destination. The hash signatures are created on both sides of a network at the point where data is local, and the hash database consists of hash signatures of blocks of data that are stored locally. The hash signatures are created using different traversal patterns across local data so that the hash database can represent a larger dataset then the actual physical storage of the local data. If no local data is present, then arbitrary data is generated and then remains static.

Claims

1. A method for data transfer comprising:

generating a first hash value of a first data chunk based on a first traversal order of the first data chunk;

generating a second hash value of the first data chunk based on a second traversal order of the first data chunk;

storing the first hash value and a specification of the first traversal order in a data store;

storing the second hash value and a specification of the second traversal order in the data store;

receiving a third hash value;

determining that the third hash value is present in the data store;

determining a third data chunk corresponding to the third hash value based on a third traversal order in the data store.

2. The method of claim 1, further comprising:

generating a plurality of data chunks, the plurality of data chunks comprising the first data chunk.

3. The method of claim 2, wherein generating the plurality of data chunks randomly generating data.

4. The method of claim 2, further comprising:

storing the plurality of data chunks in a computer readable medium.

5. The method of claim 1, wherein determining the third data chunk comprises creating the third data chunk by applying the third traversal order to the data store.

6. The method of claim 1, wherein the data store is a database.

7. The method of claim 1, wherein the third hash value is received via a network.

8. The method of claim 1, further comprising:

receiving a batch of hash values, the batch comprising the third hash value.

9. The method of claim 1, further comprising:

receiving a fourth hash value;

determining that the fourth hash value is not present in the data store;

requesting a fourth data chunk corresponding to the fourth hash value via a network.

10. A computer program product for data transfer, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

generate a first hash value of a first data chunk based on a first traversal order of the first data chunk;

generate a second hash value of the first data chunk based on a second traversal order of the first data chunk;

store the first hash value and a specification of the first traversal order in a data store;

store the second hash value and a specification of the second traversal order in the data store;

receive a third hash value;

determine that the third hash value is present in the data store;

determine a third data chunk corresponding to the third hash value based on a third traversal order in the data store.