Enhancing Analytics Performance Using Distributed Multi-Tiering

• US 20140173199A1
• Filed: 12/14/2012
• Published: 06/19/2014
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gupta, Karan, Pucha, Himabindu, Sarkar, Prasenjit
• Granted Patent: US 9,021,219 B2
• Time in Patent Office: Days
• US Class Current: 711/117
• CPC Class Codes: G06F 12/08 in hierarchically structured memory systems, e.g. virtual memory systems
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11
• CPC Class Codes: G06F 2212/254 Distributed memory

Abstract

Embodiments of the invention relate to cluster-centric tiered storage with a flexible tier definition to support performance of transactions. Object data is distributed in a multi-tiered shared-nothing cluster. Hierarchical tiers of data storage are assigned different roles within the hierarchy. The tiers are managed globally across the cluster and objects are placed in tiers according to a flexible tier definition. The probability of object access is computed for objects, and objects are moved to different tiers responsive to the computation to minimize system runtime. The location of an object is further optimized in response to an access request.

Claims

1. A method comprising:

in a computer cluster in communication with a storage system, the storage system organized in a hierarchy, assigning a different role to each storage tier within the hierarchy, each role related to a workload characteristic;

managing the tiers across the cluster, the management responsive to a flexible tier definition within the hierarchy;

calculating probability of access to a data object, selecting a tier for placement of the data object based upon the calculation, and placing the data object in one of the storage tiers in the hierarchy responsive to the flexible tier definition; and

computing placement of a new data object within the hierarchy to minimize system runtime, including optimizing placement of the new data object in response to an access request such that I/O response time is optimized.

2. The method of claim 1, further comprising selecting an instance of the object within the hierarchy having a lowest predicted latency in response to a read transaction miss, the selection comprising:

in response to a read transaction for the object, searching for an instance of the object within the hierarchy;

computing a predicted fetch latency for the instance of the object; and

selecting one instance of the object within the hierarchy with a minimum computed fetch latency.

3. The method of claim 2, wherein the search for the object instance within the hierarchy is in order of tier ranking, the order selected from the group consisting of:

ascending and descending.

4. The method of claim 1, further comprising staged demotion of an object within the hierarchy, including:

reviewing an access pattern of a selected object and moving the selected object within the storage system responsive to the hierarchy, the movement selected from the group consisting of;

eviction and demotion.

5. The method of claim 4, further comprising demoting the selected object from a current tier within the hierarchy if the reviewed access pattern is below a threshold value.

6. The method of claim 4, further comprising evicting the selected object from a current tier local to a first node in the cluster to an equivalent tier local to a second node in the cluster.

7. The method of claim 1, further comprising determining movement of a replica of the data object between tiers within a distributed global clock algorithm.

8. The method of claim 1, further comprising in response to a read transaction, reviewing a reference pattern of the object, and moving the object within the hierarchy responsive to the access pattern.

9. A computer program product for use with objects in a computer cluster, the computer program product comprising a computer-readable storage medium having program code embodied therewith, the program code being executable by a processor to:

organize a storage system in communication with the cluster in a hierarchy, including assigning a different role to each storage tier within the hierarchy, each role related to a workload characteristic;

manage the tiers across the cluster, responsive to a flexible tier definition within the hierarchy;

calculate probability of access to a data object, select a tier for placement of the data object, the selection based on the calculation, and place the data object in one of the storage tiers in the hierarchy in response to the flexible tier definition; and

compute placement of a new data within the hierarchy to minimize system runtime, including optimizing the location of the new object in response to an access request.

10. The computer program product of claim 9, further comprising computer program code to select an instance of the object within the hierarchy having a lowest predicted latency in response to a read transaction miss, the selection comprising program code to:

search for an instance of the object within the hierarchy in response to a read transaction for the object;

compute a predicted fetch latency for the instance of the object; and

select one instance of the object within the hierarchy with a minimum computed fetch latency.

11. The computer program product of claim 9, further comprising computer program code to implement staged demotion of an object within the cluster, including:

review an access pattern of a selected object and move the selected object within the storage system responsive to the hierarchy, the movement selected from the group consisting of;

eviction and demotion.

12. The computer program product of claim 11, further comprising computer program code to demote the selected object from a current tier within the hierarchy if the reviewed access pattern is below a threshold value.

13. The computer program product of claim 11, further comprising computer program code to evict the selected object from a current tier local to a first node in the cluster to an equivalent tier local to a second node in the cluster.

14. The computer program product of claim 9, further comprising computer program code to determine movement of a replica of the data object between tiers within a distributed global clock algorithm.

15. The computer program product of claim 9, further comprising computer program code to review a reference pattern of the object and move the object within the hierarchy responsive to the access pattern, in response to a read transaction.

16. A system comprising:

two or more nodes in a computer cluster in communication with a storage system, the storage system organized in a hierarchy and a flexible tier definition within the hierarchy, each tier assigned a different role within the hierarchy;

a filesystem in communication with the storage system, the file system to manage the tiers across the cluster responsive to the flexible tier definition, including;

placement of one or more objects in a select tier in the hierarchy responsive to the flexible tier definition; and

compute new data placement within the hierarchy to minimize system runtime, including to optimize the location of the new object in response to an access request.

17. The system of claim 16, further comprising the filesystem to select an instance of the object within the hierarchy having a lowest predicted latency in response to a read transaction miss, the selection comprising:

in response to a read transaction for the object, search for an instance of the object within the hierarchy;

compute a predicted fetch latency for the instance of the object; and

select one instance of the object with a minimum computed fetch latency.

18. The system of claim 17, further comprising the filesystem to demote an object within the cluster, including:

review an access pattern of a selected object and move the selected object within the hierarchy, the movement selected from the group consisting of;

eviction and demotion.

19. The system of claim 18, further comprising the filesystem to evict the selected object from a current tier local to a first node to an equivalent tier local to a second node, the first and second node in the cluster.

20. The system of claim 17, further comprising the filesystem to determine movement of a replica of the data object between tiers within a distributed global clock algorithm.

21. A method comprising:

in a computer cluster in communication with a storage system organized in a hierarchy, assigning different roles to each storage tier within the hierarchy;

placing one or more objects in a select storage tier in the hierarchy responsive to a flexible tier definition; and

managing the tiers globally across the cluster, including one or more objects assigned to a tier, the management responsive to a flexible tier definition within the hierarchy, including computing data placement within the hierarchy to minimize system runtime, and calculating probability of access to the object and optimizing the location of the object in response to an access request.

22. The method of claim 21, further comprising placing object data within a tier responsive to the calculated probability of access to the object.

23. The method of claim 22, further comprising moving a select object within the hierarchy responsive to the calculated probability of access to the select object.

24. The method of claim 23, wherein movement of the object includes inter-tier movement and intra-tier movement.

25. The method of claim 22, further comprising moving a select object within the cluster responsive to the calculated probability of access to the select object.