File Transfer Using Standard Blocks and Standard-Block Identifiers

• US 20100057750A1
• Filed: 03/06/2009
• Published: 03/04/2010
• Est. Priority Date: 09/04/2008
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: AMARASINGHE, Saman P., GARNETT, Timothy, AASTED, Matthew, SHAH, Meera
• Granted Patent: US 9,235,577 B2
• Time in Patent Office: Days
• US Class Current: N/A
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

Instead of transferring a large original file, such as a virtual-machine image file, from a source system to a target system, the original file is encoded to define a recipe file that is transferred. The recipe is then decoded to yield a duplicate of the original file on the target system. Encoding involves identifying standard blocks in the original file and including standard-block identifiers for the standard blocks in the recipe in lieu of the original blocks. Decoding involves an exchange with a standard-block identifier server system, which provides standard blocks in response to received standard-block identifiers.

Claims

1. A system comprising:

a SBID-to-S-block converter for returning over a network to a target system a version of a standard data block in response to a respective standard-block identifier received over said network from said target system, said SBID-block converter storing versions of plural standard data blocks and standard hashes generated from respective ones of said standard data blocks, said SBID-block converter associating respective standard-block identifiers with respective ones of said standard data blocks; and

a hash-to-SBID converter for returning over said network to a source system a first standard-block identifier for a first of said standard data blocks in response to a reception by said hash-to-SBID converter from said source system of a first file hash matching a first standard hash generated from said first standard data block.

2. A system as recited in claim 1 wherein said hash-SBID converter, in response to receiving from said source system a second file hash that is not identical to any of said standard data blocks, returns an indication that said second file hash does not correspond to any of said standard data blocks.

3. A system as recited in claim 2 further comprising said source system and said target system, said source system including an encoder for generating a recipe file by encoding an original file, said target system including a decoder generating a duplicate of said original file by decoding said recipe file, said recipe file including said first standard-block identifier and a version of a first file block of said original file.

4. A system as recited in claim 3 wherein said source system includes a chunker for dividing said original file into file blocks, a hash generator for generating file hashes from said file blocks, and a hash table for associating said file hashes with respective file-block identifiers associated with said file blocks, said encoder

in the event a currently-processed file hash matches a previously processed file hash, associating in said recipe file a file-block identifier for said currently processed file block with a file-block identifier for said previously processed file hash, and

in the event a currently-processed file hash does not match a previously processed file hash, adding an entry in said hash table associated with said currently processed file hash and its file-block identifier in said hash table and transmitting the currently-processed file hash to said hash-UID converter.

5. A system as recited in claim 4 wherein said first input hash is not transmitted to said hash-SBID converter.

6. A system as recited in claim 1 wherein said original file is a virtual-machine image file.

7. A method comprising:

converting on a source system an original file into original blocks;

generating file hashes from said original blocks;

transmitting some of said file hashes over a network to a SBID server system;

distinguishing file hashes that match standard hashes generated from standard blocks from those that do not match said standard hashes;

for each of said file hatches that matches a standard hash, returning a respective standard-block identifier to said source system; and

generating a recipe file including said standard-block identifier;

transferring over said network said recipe file to a target system.

8. A method as recited claim 7 further comprising:

for each of said file hashes that does not match a standard hash, returning to said source system an indication that that said file hash does not match a standard hash; and

for each of said file hashes that does not match a standard hash and not for any file hash that matches a standard hash, including in said recipe file a compressed version of the original block from which said file hash was generated.

9. A method as recited in claim 8 further comprising:

determining whether a currently processed file hash matches a previously processed file hash, and

if so, associating in said recipe file a file-block identifier for said currently-processed file hash with a file-block identifier for said previously-processed file hash, and not transmitting said currently-processed file hash to said SBID server system, and

if not, transmitting said currently processed file hash to said SBID server system.

10. A method as recited in claim 9 further comprising decoding said recipe file to yield a duplicate file of said original file on said target system.

11. A method as recited in claim 10 wherein said decoding involves:

transmitting SBIDs in said recipe file from said target system over said network to said SBID server system; and

returning versions of said standard blocks from said SBID server system over said network to said target system.

12. A method as recited in claim 7 wherein said original file is a virtual-machine image file.

13. A manufacture comprising computer-readable storage media encoded with standard data blocks and programs of computer-executable instructions, said programs providing for:

dividing an original file into uncompressed file blocks;

compressing said uncompressed file blocks to yield compressed file blocks;

generating file hashes from said compressed file blocks;

for each of said file hashes, as it becomes a currently-processed file hash, determining whether it matches a previously-processed file hash represented in an entry in a table that associates previously-processed file hashes with file-block identifiers;

in the event a currently-processed file hash matches a previously-processed file hash, associating a file-block identifier for the currently-processed file hash with a file-block identifier for the matching previously-processed file hash, and

in the event a currently-processed file hash does not match a previously-processed file hash, transmitting the currently-processed file hash to a SBID server system and, if a SBID indicting that that hash corresponds to standard block is returned from said SBID server system, including that SBID in said recipe file.

14. A manufacture as recited in claim 13 wherein, if said SBID system returns an indication that said currently-processed file hash does not correspond to a standard block, including a copy of a compressed file block from which said currently processed hash was generated in said recipe file.

15. A manufacture as recited in claim 13 wherein said programs further provide for:

transmitting said recipe file from a source system to a target system; and

decoding said recipe file to yield a duplicate of said original file.

16. A manufacture as recited in claim 15 wherein said decoding involves transmitting a SBID in said recipe file to said SBID server system and incorporating a version of a standard block received in return in said duplicate file.

17. A manufacture as recited in claim 16 wherein said decoding involves decompressing said copy of said compressed block in said recipe file and including the resulting expanded block in said duplicate file.

18. A manufacture as recited in claim 13 wherein said original file is a virtual-machine image file.