Efficient content meta-data collection and trace generation from deduplicated storage

• US 8,667,032 B1
• Filed: 12/22/2011
• Issued: 03/04/2014
• Est. Priority Date: 12/22/2011
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Shilane, Philip, Wallace, Grant, Douglis, Frederick
• Primary Examiner(s): MORRISON, JAY A
• Application Number:
  US13/335,750
• Time in Patent Office: 803 Days
• Field of Search: None
• US Class Current: 707/825
• CPC Class Codes: G06F 11/3034 where the computing system component is a storage system, e.g. DASD based or network based digital input from or digital output to record carriers G06F3/06; digital recording or reproducing G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 2201/83 the solution involving signatures

Abstract

The method and apparatus collect file recipes from deduplicated data storage systems, the file recipes consist of a list of fingerprints of data chunks of a file. Detailed meta-data for each unique data chunk is also collected. In an offline process, research and analysis can be performed on either the meta-data itself or on a reconstruction of a full trace of meta-data constructed by matching recipe fingerprints to the corresponding meta-data. The method and system can generate the full meta-data trace efficiently in an on-line or off-line process. Typical deduplicated storage systems achieve 10× or higher deduplication rates, and the meta-data collection is faster than processing all of the original files and produces compact meta-data that is smaller to store.

Claims

1. A computer-implemented method for collecting meta-data from a deduplication data storage system, the method comprising:

collecting a set of file recipes for a set of files stored in the deduplication data storage system, each file recipe in the set of file recipes including a fingerprint for each unique data chunk that constitutes a file, wherein each fingerprint identifies each corresponding unique data chunk;

collecting meta-data for a set of unique data chunks for the collected set of files by a data collection engine, wherein the meta-data describes the unique data chunks;

anonymizing the collected set of file recipes and the meta-data by an anonymizing engine; and

storing the anonymized set of file recipes and the anonymized meta-data in a data collection storage unit for content data set analysis without the content data set.

2. The method of claim 1, wherein the meta-data includes any one of data chunk type, local compression size, sub-chunk sizes, fingerprint, data chunk hashes, data chunk resemblance hashes for delta encoding and disk location.

3. The method of claim 1, further comprising:

dividing each file in the set of files into a set of data chunks and storing corresponding file recipe;

generating a fingerprint for each data chunk in the set of data chunks; and

checking each fingerprint against an index to determine unique data chunks to store associated meta-data.

4. The method of claim 1, wherein the collecting the set of file recipes and the collecting the meta-data are part of an on-line process that occurs concurrently with a data storage process for the set of files.

5. The method of claim 1, wherein the collecting the set of file recipes and the collecting the meta-data are part of an off-line process that occurs after the data storage process for the set of files.

6. The method of claim 1, wherein the set of files is greater than five terabytes.

7. A non-transitory computer-readable storage medium having instructions stored therein, which when executed by a computer, cause the computer to perform a method, the method for collecting meta-data from a deduplication data storage system, the method comprising:

collecting a set of file recipes for a set of files stored in the deduplication data storage system, each file recipe in the set of file recipes including a fingerprint for each unique data chunk that constitutes a file, wherein each fingerprint identifies each corresponding unique data chunk;

collecting meta-data for a set of unique data chunks for the collected set of files by a data collection engine, wherein the meta-data describes the unique data chunks;

anonymizing the collected set of file recipes and the meta-data by an anonymizing engine; and

storing the anonymized set of file recipes and the anonymized meta-data in a data collection storage unit for content data set analysis without the content data set.

8. The non-transitory computer-readable storage medium of claim 7, wherein the meta-data includes any one of data chunk type, local compression size, sub-chunk sizes, fingerprint, data chunk hashes, data chunk resemblance hashes for delta encoding and disk location.

9. The non-transitory computer-readable storage medium of claim 7, further comprising:

dividing each file in the set of files into a set of data chunks and storing corresponding file recipe;

generating a fingerprint for each data chunk in the set of data chunks; and

checking each fingerprint against an index to determine unique data chunks to store associated meta-data.

10. The non-transitory computer-readable storage medium of claim 7, wherein the collecting the set of file recipes and the collecting the meta-data are part of an on-line process that occurs concurrently with a data storage process for the set of files.

11. The non-transitory computer-readable storage medium of claim 7, wherein the collecting the set of file recipes and the collecting the meta-data are part of an off-line process that occurs after the data storage process for the set of files.

12. The non-transitory computer-readable storage medium of claim 7, wherein the set of files is greater than five terabytes.

13. A deduplication storage system, comprising:

a processing system configuring to execute (1) a data collection engine to collect a set of file recipes for a set of files stored in the deduplication data storage system, each file recipe in the set of file recipes including a fingerprint for each unique data chunk that constitutes a file, wherein each fingerprint identifies each corresponding unique data chunk, the data collection engine to collect meta-data for a set of unique data chunks for the collected set of files, wherein the meta-data describes the unique data chunks and (2) an anonymizing engine that is communicatively coupled to the data collection engine, the anonymizing engine to anonymize the collected set of file recipes and the meta-data; and

a data collection storage unit communicatively coupled to the processing system and the data collection engine, the data collection storage unit to store the anonymized set of file recipes and the anonymized meta-data for content data set analysis without the content data set.

14. The storage system of claim 13, wherein the meta-data includes any one of data chunk type, local compression size, sub-chunk sizes, fingerprint, data chunk hashes, data chunk resemblance hashes for delta encoding and disk location.

15. The storage system of claim 13, wherein the processing system is further configured to execute:

a data chunk storage engine that is coupled to the data collection engine, the data chunk storage engine to divide each file in the set of files into a set of data chunks.

16. The storage system of claim 15, wherein the data chunk storage engine generates a fingerprint for each data chunk in the set of data chunks.

17. The storage system of claim 16, wherein the data chunk storage engine checks each fingerprint against an index to determine unique data chunks.