Data deduplication by separating data from meta data

• US 7,962,452 B2
• Filed: 12/28/2007
• Issued: 06/14/2011
• Est. Priority Date: 12/28/2007
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Anglin, Matthew Joseph
• Primary Examiner(s): AL HASHEMI, SANA A
• Application Number:
  US11/966,460
• Publication Number:
  US 20090171888A1
• Time in Patent Office: 1,264 Days
• Field of Search: 707/634, 707/645, 707/640, 707/729, 707/791
• US Class Current: 707/645
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641

Abstract

Provided are techniques for data deduplication. A chunk of data and a mapping of boundaries between file data and meta data in the chunk of data are received. The mapping is used to split the chunk of data into a file data stream and a meta data stream and to store file data from the file data stream in a first file and to store meta data from the meta data stream in a second file, wherein the first file and the second file are separate files. The file data in the first file is deduplicated.

Claims

1. A method, comprising:

inserting meta data into a file containing file data to form co-mingled data;

sending chunks of the co-mingled data and a mapping of boundaries that identifies offsets of which portions of each chunk of data are the meta data and which portions of each chunk of data are the file data to a data deduplication system; and

under control of the data deduplication system,

receiving a chunk of data storing the co-mingled file data and meta data and the mapping;

using the mapping to split the chunk of data into a file data stream and a meta data stream and to store the file data from the file data stream in a first file and to store the meta data from the meta data stream in a second file, wherein the first file and the second file are separate files; and

deduplicating the file data in the first file without deduplicating the meta data in the second file.

2. The method of claim 1, further comprising:

storing header information with each portion of the chunk of data that is meta data, wherein the header information includes an offset into the file data at which the meta data is to be inserted.

3. The method of claim 1, further comprising:

reassembling the chunk of data by;

alternatively reading the meta data and the file data based on header information associated with the meta data; and

streaming the meta data and the file data to a data buffer, wherein the meta data is inserted into the file data at an offset indicated by the header information associated with the meta data.

4. The method of claim 3, further comprising:

reconstructing the file data by accessing locations of chunks of data.

5. The method of claim 1, further comprising:

performing two levels of reconstruction, wherein a first level of reconstruction reconstructs a deduplicated file, and wherein a second level of reconstruction re-inserts meta data into file data.

6. The method of claim 1, further comprising:

while receiving a buffer of data that includes co-mingled file data and meta data describing the file data, generating the mapping of boundaries between the file data and the meta data by keeping track of which portions of the buffer of data are file data and which portions of the buffer of data are meta data; and

forwarding the buffer of data as the chunk of data to a data deduplication system along with the mapping.

7. A computer program product comprising a computer-readable medium including computer readable instructions, wherein the computer readable instructions when executed by a processor on a computer causes the computer to:

insert meta data into a file containing file data to form co-mingled data;

send chunks of the co-mingled data and a mapping of boundaries that identifies offsets of which portions of each chunk of data are the meta data and which portions of each chunk of data are the file data to a data deduplication system; and

under control of the data deduplication system,

receive a chunk of data storing the co-mingled file data and meta data and the mapping;

use the mapping to split the chunk of data into a file data stream and a meta data stream and to store the file data from the file data stream in a first file and to store the meta data from the meta data stream in a second file, wherein the first file and the second file are separate files; and

deduplicate the file data in the first file without deduplicating the meta data in the second file.

8. The computer program product of claim 7, wherein the computer readable program when executed on a computer causes the computer to:

store header information with each portion of the chunk of data that is meta data, wherein the header information includes an offset into the file data at which the meta data is to be inserted.

9. The computer program product of claim 7, wherein the computer readable program when executed on a computer causes the computer to:

reassemble the chunk of data by;

alternatively reading the meta data and the file data based on header information associated with the meta data; and

streaming the meta data and the file data to a data buffer, wherein the meta data is inserted into the file data at an offset indicated by the header information associated with the meta data.

10. The computer program product of claim 9, wherein the computer readable program when executed on a computer causes the computer to:

reconstruct the file data by accessing locations of chunks of data.

11. The computer program product of claim 7, wherein the computer readable program when executed on a computer causes the computer to:

perform two levels of reconstruction, wherein a first level of reconstruction reconstructs a deduplicated file, and wherein a second level of reconstruction re-inserts meta data into file data.

12. The computer program product of claim 7, wherein the computer readable program when executed on a computer causes the computer to:

while receiving a buffer of data that includes co-mingled file data and meta data describing the file data, generate the mapping of boundaries between the file data and the meta data by keeping track of which portions of the buffer of data are file data and which portions of the buffer of data are meta data; and

forward the buffer of data as the chunk of data to a data deduplication system along with the mapping.

13. A system, comprising:

hardware logic performing operations, the operations comprising;

inserting meta data into a file containing file data to form co-mingled data;

sending chunks of the co-mingled data and a mapping of boundaries that identifies offsets of which portions of each chunk of data are the meta data and which portions of each chunk of data are the file data to a data deduplication system; and

hardware logic of the data deduplication system performing operations, the operations comprising;

receiving a chunk of data storing the co-mingled file data and meta data and the mapping;

using the mapping to split the chunk of data into a file data stream and a meta data stream and to store the file data from the file data stream in a first file and to store the meta data from the meta data stream in a second file, wherein the first file and the second file are separate files; and

deduplicating the file data in the first file without deduplicating the meta data in the second file.

14. The system of claim 13, wherein the operations further comprise:

storing header information with each portion of the chunk of data that is meta data, wherein the header information includes an offset into the file data at which the meta data is to be inserted.

15. The system of claim 13, wherein the operations further comprise:

reassembling the chunk of data by;

alternatively reading the meta data and the file data based on header information associated with the meta data; and

streaming the meta data and the file data to a data buffer, wherein the meta data is inserted into the file data at an offset indicated by the header information associated with the meta data.

16. The system of claim 15, wherein the operations further comprise:

reconstructing the file data by accessing locations of chunks of data.

17. The system of claim 13, wherein the operations further comprise:

performing two levels of reconstruction, wherein a first level of reconstruction reconstructs a deduplicated file, and wherein a second level of reconstruction re-inserts meta data into file data.