System and method for data deduplication

• US 8,205,065 B2
• Filed: 03/30/2009
• Issued: 06/19/2012
• Est. Priority Date: 03/30/2009
• Status: Active Grant

• Current Assignee: Exar Corporation (Maxlinear Incorporated)
• Original Assignee: Exar Corporation (Maxlinear Incorporated)
• Inventors: Matze, John Edward Gerard
• Primary Examiner(s): CHOE, YONG J
• Application Number:
  US12/414,538
• Publication Number:
  US 20100250896A1
• Time in Patent Office: 1,177 Days
• Field of Search: 711/154, 711/216, 711/112, 711/115, 711/E12.001
• US Class Current: 711/216
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0674 Disk device

Abstract

A system for deduplicating data comprises a card operable to receive at least one data block and a processor on the card that generates a hash for each data block. The system further comprises a first module that determines a processing status for the hash and a second module that discards duplicate hashes and their data blocks and writes unique hashes and their data blocks to a computer readable medium. In one embodiment, the processor also compresses each data block using a compression algorithm.

Claims

1. A system for deduplicating data comprising:

a computer that includes a computer readable medium and general processor operating thereon;

a card, coupled to the computer, that includes a processor for performing hashing operations and hardware-assisted compression;

a virtual block device that is an interface with applications and that is operable to set a block size for deduplication;

a deduplication block manager that passes blocks to the processor and receives a hash for each block and a compressed block from the processor;

a truncated hash array that includes an index for each possible truncated hash value and a corresponding entry initially set to a default value wherein when a new hash is processed, the corresponding entry is modified to include a pointer to the new hash and block;

wherein the card

receives blocks from the deduplication block manager,

creates a hash value for each block of data using a hashing algorithm, and

compresses each block of data using a compression algorithm; and

wherein the deduplication block manager

receives the hash values and the compressed blocks from the card, and

creates a truncated hash corresponding to each hash value.

2. The system of claim 1 further comprising:

wherein the deduplication block manager looks up each truncated hash in the truncated hash array; and

wherein if the corresponding entry to the truncated hash is the default value then the virtual block device writes the compressed block to a computer readable medium and adds a pointer to the compressed block and hash value to the truncated hash array.

3. The system of claim 1 further comprising:

wherein the deduplication block manager looks up each truncated hash in the truncated hash array; and

wherein if the corresponding entry to the truncated hash includes a pointer to a processed hash value and compressed block then the deduplication block manager retrieves the processed hash value and checks whether the hash value corresponding to the truncated hash matches the processed hash value.

4. The system of claim 3 wherein if the hash value matches the processed hash value then the compressed block is discarded.

5. The system of claim 3 wherein if the hash value does not match the processed hash value then the deduplication block manager determines if there is are pointers to additional processed hash values and the additional processed hash values are compared with the hash value.

6. The system of claim 5 wherein if the hash value does not match any processed hash values then the virtual block device writes the compressed block to a computer readable medium and adds a pointer to the compressed block and hash value to the truncated hash array.

7. The system of claim 1, wherein virtual cluster locations are configured to align with hash blocks.

8. The system of claim 1, wherein the virtual block device works with the blocks, and has no knowledge of any file or directory structure.