DE-DUPLICATION DEPLOYMENT PLANNING

• US 20140358870A1
• Filed: 09/03/2013
• Published: 12/04/2014
• Est. Priority Date: 06/03/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Chambliss, David D., Constantinescu, Mihail C., Glider, Joseph S., Lu, Maohua
• Granted Patent: US 9,275,068 B2
• Time in Patent Office: Days
• US Class Current: 707/692
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641

Abstract

Assignment of files to a de-duplication domain. Address space of data files is divided into multiple containers. For each of the containers, a file metadata scan is performed to obtain file system metadata, which is aggregated and summarized in a content feature summary. A content feature summary prediction measurement is measured between containers from the generated content feature summary, and files from each container are assigned to a de-duplication domain based upon the content similarity predication measurement.

Claims

1. A method comprising:

dividing an address space of files into multiple containers;

performing a file metadata scan, including obtaining attributes for files in each container;

aggregating the file attributes into characterizations for each attribute dimension, and generating a content feature summary for each container incorporating the characterizations;

measuring a content similarity prediction measurement between containers from the generated content feature summary; and

assigning files from each container to a de-duplication domain based on the computed content similarity prediction measurement.

2. The method of claim 1, wherein the file attributes include file system metadata.

3. The method of claim 2, further comprising estimating a discrete file similarity through use of one or more discrete file summaries, each discrete file summary including a list of records, each record corresponding to one file, and each record having a signature computed from a file name and a file size.

4. The method of claim 2, further comprising computing an owner group distribution similarity, including comparing owner group distributions for two or more containers and computing a correlation between the groups.

5. The method of claim 4, wherein the computation of the owner group distribution similarity includes determining a set of owner groups, processing metadata for files in a container, and associating each file with one of the owner groups, and computing the owner group distribution.

6. The method of claim 2, further comprising computing type and size distributions within containers, comparing said distributions between containers, and computing a similarity between containers based on the comparison.

7. The method of claim 1, wherein the assignment of files includes placing containers with high similarity into a common de-duplication domain.