Data deduplication by separating data from meta data

• US 8,055,618 B2
• Filed: 04/18/2011
• Issued: 11/08/2011
• Est. Priority Date: 12/28/2007
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Anglin, Matthew J.
• Primary Examiner(s): AL HASHEMI, SANA A
• Application Number:
  US13/088,875
• Publication Number:
  US 20110196848A1
• Time in Patent Office: 204 Days
• Field of Search: 707/634, 707/638, 707/791, 707/792
• US Class Current: 707/634
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641

Abstract

Provided are techniques for data deduplication. A chunk of data and a mapping of boundaries between file data and meta data in the chunk of data are received. The mapping is used to split the chunk of data into a file data stream and a meta data stream and to store file data from the file data stream in a first file and to store meta data from the meta data stream in a second file, wherein the first file and the second file are separate files. The file data in the first file is deduplicated.

Claims

1. A method, comprising:

under control of a data deduplication system,

receiving chunks of co-mingled data having file data and meta data;

splitting the chunks of data into a file data stream stored in a first file and a meta data stream stored in a second file; and

deduplicating the file data in the first file without deduplicating the meta data in the second file; and

in response to receiving a request for the chunks of data,

alternating between reading the meta data stream and the file data stream to fill a data buffer by;

determining whether to read the meta data stream or the file data stream next based on header information in the meta data;

in response to determining that the meta data stream is to be read, reading the meta data into the data buffer; and

in response to determining that the file data stream is to be read, reading the file data into the data buffer while reassembling the file data that was previously deduplicated using a reconstruction structure to put together the chunks of the file data stream; and

returning the chunks of co-mingled data having the file data and the meta data in the data buffer.

2. The method of claim 1, further comprising:

reading the file data into the data buffer until running out of file data, filling the data buffer, or reaching a next meta data insertion point.

3. The method of claim 1, wherein the header information includes an offset into the file data at which the meta data is to be inserted and further comprising:

inserting the meta data into the file data at the offset.

4. The method of claim 1, wherein the reconstruction structure comprises a table that includes a column for file names and a column for chunks of files and locations of those chunks.

5. The method of claim 1, further comprising:

performing two levels of reconstruction by reconstructing a deduplicated file at a first level and re-inserting the meta data into the file data at a second level.

6. The method of claim 1, further comprising:

inserting the meta data into a file containing the file data to form the co-mingled data; and

sending the chunks of the co-mingled data to the data deduplication system.

7. A computer program product comprising a computer-readable medium including computer readable instructions, wherein the computer readable instructions, when executed by a processor on a computer, causes the computer to:

under control of a data deduplication system,

receive chunks of co-mingled data having file data and meta data;

split the chunks of data into a file data stream stored in a first file and a meta data stream stored in a second file; and

deduplicate the file data in the first file without deduplicating the meta data in the second file; and

in response to receiving a request for the chunks of data,

alternate between reading the meta data stream and the file data stream to fill a data buffer by;

determining whether to read the meta data stream or the file data stream next based on header information in the meta data;

in response to determining that the meta data stream is to be read, reading the meta data into the data buffer; and

in response to determining that the file data stream is to be read, reading the file data into the data buffer while reassembling the file data that was previously deduplicated using a reconstruction structure to put together the chunks of the file data stream; and

return the chunks of co-mingled data having the file data and the meta data in the data buffer.

8. The computer program product of claim 7, wherein the computer readable instructions, when executed by the processor on the computer, causes the computer to:

read the file data into the data buffer until running out of file data, filling the data buffer, or reaching a next meta data insertion point.

9. The computer program product of claim 7, wherein the header information includes an offset into the file data at which the meta data is to be inserted and wherein the computer readable instructions, when executed by the processor on the computer, causes the computer to:

insert the meta data into the file data at the offset.

10. The computer program product of claim 7, wherein the reconstruction structure comprises a table that includes a column for file names and a column for chunks of files and locations of those chunks.

11. The computer program product of claim 7, wherein the computer readable instructions, when executed by the processor on the computer, causes the computer to:

perform two levels of reconstruction by reconstructing a deduplicated file at a first level and re-inserting the meta data into the file data at a second level.

12. The computer program product of claim 7, wherein the computer readable instructions, when executed by the processor on the computer, causes the computer to:

insert the meta data into a file containing the file data to form the co-mingled data; and

send the chunks of the co-mingled data to the data deduplication system.

13. A system, comprising:

hardware logic of a data deduplication system performing operations, the operations comprising;

receiving chunks of co-mingled data having file data and meta data;

splitting the chunks of data into a file data stream stored in a first file and a meta data stream stored in a second file; and

deduplicating the file data in the first file without deduplicating the meta data in the second file; and

in response to receiving a request for the chunks of data,

alternating between reading the meta data stream and the file data stream to fill a data buffer by;

determining whether to read the meta data stream or the file data stream next based on header information in the meta data;

in response to determining that the meta data stream is to be read, reading the meta data into the data buffer; and

in response to determining that the file data stream is to be read, reading the file data into the data buffer while reassembling the file data that was previously deduplicated using a reconstruction structure to put together the chunks of the file data stream; and

returning the chunks of co-mingled data having the file data and the meta data in the data buffer.

14. The system of claim 13 wherein the operations further comprise:

reading the file data into the data buffer until running out of file data, filling the data buffer, or reaching a next meta data insertion point.

15. The system of claim 13, wherein the header information includes an offset into the file data at which the meta data is to be inserted and wherein the operations further comprise:

inserting the meta data into the file data at the offset.

16. The system of claim 13, wherein the reconstruction structure comprises a table that includes a column for file names and a column for chunks of files and locations of those chunks.

17. The system of claim 13, wherein the operations further comprise:

performing two levels of reconstruction by reconstructing a deduplicated file at a first level and re-inserting the meta data into the file data at a second level.

18. The system of claim 13, wherein the operations further comprise:

inserting the meta data into a file containing the file data to form the co-mingled data; and

sending the chunks of the co-mingled data to the data deduplication system.