Packing deduplicated data in a self-contained deduplicated repository

• US 9,678,972 B2
• Filed: 11/07/2013
• Issued: 06/13/2017
• Est. Priority Date: 01/10/2013
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hirsch, Michael, Leneman, Ofer, Akirav, Shay H.
• Primary Examiner(s): Alam, Hosain
• Assistant Examiner(s): May, Robert F
• Application Number:
  US14/074,195
• Publication Number:
  US 20140195495A1
• Time in Patent Office: 1,314 Days
• Field of Search: 707692, 711162
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Deduplicated data is packed in a self-contained deduplicated repository having unique data blocks with each being referenced by a globally unique identifier (GUID). The self-contained deduplicated repository has information regarding both deduplicated data files and the unique data blocks of each of the deduplicated data files and a master GUID list containing a location of each of the unique data blocks.

Claims

1. A method for storing deduplicated data using a processor device in a computing environment, the method comprising:

packing deduplicated data in a self-contained deduplicated repository having a plurality of unique data blocks with each being referenced by a globally unique identifier (GUID), and the self-contained deduplicated repository also having information regarding both deduplicated data files and the plurality of unique data blocks of each of the deduplicated data files and a master GUID list containing a location of each of the plurality of unique data blocks;

loading the self-contained deduplicated repository into a target repository, the target repository being stored in a virtual environment;

creating the self-contained deduplicated repository from a list of deduplicated data files, wherein the self-contained deduplicated repository is isolated such that the self-contained deduplicated repository is stored at a remote location without requiring power and contains a table of contents (TOC) of the deduplicated data files, metadata of the deduplicated data files, and one unique copy of each of the plurality of unique data blocks; and

merging all deduplicated data files in the self-contained deduplicated repository into a target repository by reading the TOC and deduplication metadata, wherein the TOC and deduplication metadata are stored, and any of the GUID needed by the TOC that do not exist in the target repository are read from the self-contained deduplicated repository and stored in the target repository.

2. The method of claim 1, further including examining the TOC of each of the deduplicated data files in the list of deduplicated data files one after another, wherein the master GUID list is maintained, and the TOC is an ordered list of the GUID that make up a file.

3. The method of claim 2, further including adding to the master GUID list each one of the plurality of unique data blocks required by any one of the deduplicated data files in the list of deduplicated data files containing a location of each of the plurality of unique data blocks.

4. The method of claim 1, further including committing to the self-contained deduplicated repository the TOC and metadata of each of the deduplicated data files in the list of deduplicated data files, the master GUID list, and the plurality of unique data blocks required by any one of the deduplicated data files in the list of deduplicated data files.

5. The method of claim 1, further including performing one of:

ordering the plurality of unique data blocks in the self-contained deduplicated repository by the GUID, and

mapping a position of the plurality of unique data blocks in the self-contained deduplicated repository.

6. The method of claim 1, further including performing each of:

inputting into the self-contained deduplicated repository the list of deduplicated data files,

determining if each one of the deduplicated data files in the list of deduplicated data files are processed,

appending metadata and the TOC of the one of the deduplicated data files to the self-contained deduplicated repository if one of the deduplicated data files is not processed,

determining if each of the GUID in the one of the deduplicated data files is processed,

determining if the GUID is located in the master GUID list,

adding the GUID to the master GUID list,

writing the master GUID list to the self-contained deduplicated repository, and

writing the data of each of the GUID to the self-contained deduplicated repository.

7. The method of claim 1, further including retrieving the deduplicated data file from the self-contained deduplicated repository.

8. The method of claim 7, further including performing one of:

scanning the list of deduplicated data files at the beginning of the self-contained deduplicated repository until locating one of the deduplicated data files to be retrieved,

retrieving metadata of the one of the deduplicated data files and the TOC from the self-contained deduplicated repository,

skipping to the master GUID list and retrieving the master GUID list from the self-contained deduplicated repository,

if each of the GUID in the one of the deduplicated data files is not processed, performing each of;

finding a position of a first GUID in the one of the deduplicated data files in the master GUID list, and

adding a tuple to the master GUID list that is retrieved, and

if each of the GUID in the one of the deduplicated data files is processed, performing each of;

sorting the master GUID list that is retrieved by position,

determining if all tuples in the master GUID list have been processed,

deriving a position of the GUID of a first tuple in the self-contained deduplicated repository from the position in the master GUID list that is retrieved,

forwarding to the position of the GUID of the first tuple in the self-contained deduplicated repository and retrieving data of the GUID, and

writing the data of the GUID of the first tuple to each position in the one of the deduplicated data according to appearance in consecutive tuples in the master GUID list that is sorted.

9. The method of claim 1, further including restoring the deduplicated data file from the self-contained deduplicated repository by reading a table of contents (TOC) of the deduplicated data file and reading each one of the plurality of unique data blocks required by the deduplicated data file using the GUID listed in the TOC.

10. The method of claim 1, further including allowing each deduplication system that draws the GUID for the deduplication system from a similar GUID allocation sphere to share each one of a plurality of self-contained deduplicated repositories, extract files from each one of the plurality of self-contained deduplicated repositories, and append to each one of the plurality of self-contained deduplicated repositories.