De-Duplication Optimized Platform for Object Grouping

• US 20170344598A1
• Filed: 05/27/2016
• Published: 11/30/2017
• Est. Priority Date: 05/27/2016
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Constantinescu, M. Corneliu, Routray, Ramani R., Subratie, Kensworth C.
• Application Number:
  US15/166,890
• Publication Number:
  US 20170344598A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity

Abstract

Embodiments are provided for enhancing storage efficiency in a de-duplication enabled storage system. Using one or more de-duplication metadata repositories local to respective nodes of a storage system, objects are pre-processed in each node. The pre-processing includes deriving a coreness of each object, and grouping the objects into respective cores based on coreness. Each object of a core has at least a minimum coreness. In response to receiving an object request from a target node, the request is iteratively assessed by locating a first core comprising the requested object, calculating a size of the located first core, and identifying a transfer group based on the extracted size. The transfer group is transferred to the target node.

Claims

1. A system comprising:

one or more server nodes, each node comprising processor in communication with memory, and a local repository of de-duplication metadata;

one or more tools in communication with the nodes, the tools to;

using the repositories of de-duplication metadata, pre-process objects in each node, the pre-processing comprising the tools to derive a coreness of each object, and group the objects into respective cores based on coreness, wherein each object of a core has at least a minimum coreness;

in response to receipt of an object request from a target node, iteratively assess the request, the iterative assessment comprising the tools to;

locate a first core comprising the requested object;

calculate a size of the located first core; and

identify a transfer group based on the extracted size; and

transfer the transfer group to the target node.

2. The system of claim 1, wherein the pre-processing further comprises the one or more tools to build at least one content sharing graph, wherein each vertex of the content sharing graph corresponds to an object of the core, wherein the coreness is a weight measure associated with a quantity of total bytes shared between a vertex and its adjacent vertices, and wherein each core is a maximal connected subgraph of the content sharing graph.

3. The system of claim 2, wherein identifying the transfer group comprises the one or more tools to compare the calculated size to a maximum transfer constraint, and wherein the transfer group is identified as the selected group in response to the extracted size being less than the maximum transfer constraint.

4. The system of claim 3, further comprising the one or more tools to, in response to the calculated size exceeding the maximum transfer constraint, partition the located first core, and choose a largest byte sub-group of files from the partition, wherein the transfer group is identified as the largest byte sub-group.

5. The system of claim 3, further comprising the one or more tools to, in response to the calculated size exceeding the maximum transfer constraint, locate a second core containing the requested object, wherein the second core is associated with a higher coreness than the first core.

6. The system of claim 5, wherein the transfer group is identified as the requested object in response to a failure to select the new group.

7. A computer program product comprising a computer-readable storage medium having computer-readable program code embodied therewith, the program code executable by a processor to:

using one or more repositories of de-duplication metadata local to respective nodes of a storage system, pre-process objects in each node, the pre-processing comprising program code to derive a coreness of each object, and group the objects into respective cores based on coreness, wherein each object of a core has at least a minimum coreness;

in response to receipt of an object request from a target node, iteratively assess the request, the iterative assessment comprising program code to;

locate a first core comprising the requested object;

calculate a size of the located first core; and

identify a transfer group based on the extracted size; and

transfer the transfer group to the target node.

8. The computer program product of claim 7, wherein the pre-processing further comprises program code to build at least one content sharing graph, wherein each vertex of the content sharing graph corresponds to an object of the core, wherein the coreness is a weight measure associated with a quantity of total bytes shared between a vertex and its adjacent vertices, and wherein each core is a maximal connected subgraph of the content sharing graph.

9. The computer program product of claim 7, wherein the iterative assessment further comprises program code to compare the calculated size to a maximum transfer constraint, and wherein the transfer group is identified based on the comparison.

10. The computer program product of claim 9, wherein the transfer group is identified as the selected group in response to the calculated size being less than the maximum transfer constraint.

11. The computer program product of claim 9, further comprising program code to, in response to the calculated size exceeding the maximum transfer constraint, partition the located first core, and select a largest byte sub-group of files from the partition, wherein the transfer group is identified as the largest byte sub-group.

12. The computer program product of claim 9, further comprising program code to, in response to the calculated size exceeding the maximum transfer constraint, locate a second core containing the requested object, wherein the second core is associated with a higher coreness than the first core.

13. The computer program product of claim 12, wherein the transfer group is identified as the requested object in response to a failure to locate the second core.

14. A method comprising:

using one or more repositories of de-duplication metadata local to respective nodes of a storage system, pre-processing objects in each node, the pre-processing comprising deriving a coreness of each object, and grouping the objects into respective cores based on coreness, wherein each object of a core has at least a minimum coreness;

in response to receiving an object request from a target node, iteratively assessing the request, the iterative assessment comprising;

selecting a first core comprising the requested object;

calculating a size of the located first core; and

identifying a transfer group based on the extracted size; and

transferring the transfer group to the target node.

15. The method of claim 14, wherein the pre-processing further comprises building at least one content sharing graph, wherein each vertex of the content sharing graph corresponds to an object of the core, wherein the coreness is a weight measure associated with a quantity of total bytes shared between a vertex and its adjacent vertices, and wherein each core is a maximal connected subgraph of the content sharing graph.

16. The method of claim 14, wherein the iterative assessment further comprises comparing the calculated size to a maximum transfer constraint, and wherein the transfer group is identified based on the comparison.

17. The method of claim 16, wherein the transfer group is identified as the selected group in response to the calculated size being less than the maximum transfer constraint.

18. The method of claim 16, further comprising, in response to the calculated size exceeding the maximum transfer constraint, partitioning the located first core, and selecting a largest byte sub-group of files from the partition, wherein the transfer group is identified as the largest byte sub-group.

19. The method of claim 16, further comprising, in response to the calculated size exceeding the maximum transfer constraint, locating a second core containing the requested object, wherein the second core is associated with a higher coreness than the first core.

20. The method of claim 19, wherein the transfer group is identified as the requested object in response to a failure to locate the second core.