Multi-tier storage system having a front-end storage tier facilitating efficient graph analytics computations

• US 9,960,973 B1
• Filed: 12/22/2015
• Issued: 05/01/2018
• Est. Priority Date: 12/22/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Faibish, Sorin, Bent, John M., Ting, Dennis, Teymouri, Sassan
• Primary Examiner(s): Luu, Le H
• Application Number:
  US14/977,960
• Time in Patent Office: 861 Days
• CPC Class Codes: G06F 3/0601 Interfaces specially adapted for storage systems
• CPC Class Codes: G06F 3/0607 by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/0653 Monitoring storage devices or systems
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0685 Hybrid storage combining heterogeneous device types, e.g. hierarchical storage, hybrid arrays
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

An apparatus in one embodiment comprises a multi-tier storage system and at least one dispatcher node. The multi-tier storage system comprises at least a front-end storage tier and a back-end storage tier. The at least one dispatcher node is configured to load portions of graph data from the front-end storage tier and the back-end storage tier for graph analytics computations performed at respective ones of a plurality of compute nodes configured for communication with the storage system over at least one network. The multi-tier storage system configured to switch logical connections of respective ones of the compute nodes to the at least one dispatcher node based on progress of graph analytics computations at the compute nodes.

Claims

1. An apparatus comprising:

a multi-tier storage system comprising at least a front-end storage tier and a back-end storage tier; and

at least one dispatcher node implemented on at least one processing device comprising a processor coupled to a memory, the at least one dispatcher node configured to load portions of graph data from the front-end storage tier and the back-end storage tier for graph analytics computations performed at respective ones of a plurality of compute nodes configured for communication with the storage system over at least one network;

wherein the multi-tier storage system is configured to switch logical connections of respective ones of the compute nodes to the at least one dispatcher node based on progress of graph analytics computations at the compute nodes.

2. The apparatus of claim 1 wherein the front-end storage tier comprises a fast tier having a relatively high input-output processing speed and a relatively low capacity and the back-end storage tier comprises a capacity tier having a relatively low input-output processing speed and a relatively high capacity.

3. The apparatus of claim 2 wherein the storage system further comprises one or more data mover modules configured to move data between the front-end storage tier and the back-end storage tier.

4. The apparatus of claim 2 wherein the front-end storage tier comprises at least one server-based flash storage device and the back-end storage tier comprises at least one object store configured to store data in the form of objects having respective object identifiers.

5. The apparatus of claim 2 wherein the at least one dispatcher node is implemented as an additional tier of the multi-tier storage system above the front-end storage tier of the multi-tier storage system.

6. The apparatus of claim 1 wherein the at least one dispatcher node is implemented at least partially within a given one of the compute nodes, the at least one dispatcher node being configured to load data into a memory of the given compute node from at least one of the front-end storage tier and the back-end storage tier.

7. The apparatus of claim 1 wherein the at least one dispatcher node is implemented at least partially within the front-end storage tier of the multi-tier storage system, the at least one dispatcher node being configured to load data into the front-end storage tier from the back-end storage tier.

8. The apparatus of claim 1 wherein the compute nodes logically connected to the same dispatcher node are configured to communicate with one another while compute nodes logically connected to different dispatcher nodes are not configured to communicate with one another.

9. The apparatus of claim 8 wherein the compute nodes logically connected to the same dispatcher node are configured to communicate with one another using remote direct memory access connections.

10. The apparatus of claim 1 wherein the at least one dispatcher node comprises a flash memory configured to store a sub-graph of graph data, the sub-graph being based on the progress of graph analytics computations by compute nodes logically connected to the at least one dispatcher node.

11. The apparatus of claim 10 wherein the multi-tier storage system is configured to switch the logical connections of respective ones of the compute nodes based on hints of distance between graph nodes of the sub-graph stored in the flash memory of the at least one dispatcher node.

12. The apparatus of claim 1 wherein the graph analytics computations comprise a series of phases with each phase comprising at least one query.

13. The apparatus of claim 12 wherein the at least one dispatcher node is configured to build one or more tables for data locality detection after each query ends, the one or more tables being persisted to a flash memory of the at least one dispatcher node and utilized to predict which compute nodes will become active in a next phase.

14. The apparatus of claim 13 wherein the at least one dispatcher node utilizes the predictions to logically connect different subsets of the compute nodes to the at least one dispatcher node for each phase.

15. The apparatus of claim 1 wherein the at least one dispatcher node utilizes in-memory graphs for multi-dimensional hashing indexing middleware (MDHIM) insertions, flush and persist operations.

16. The apparatus of claim 1 wherein the multi-tier storage system and the at least one dispatcher node are part of an information processing system.

17. A method comprising:

configuring a multi-tier storage system for communication over at least one network with a plurality of compute nodes, the multi-tier storage system comprising at least a front-end storage tier and a back-end storage tier;

implementing at least one dispatcher node configured to load portions of graph data from the front-end storage tier and the back-end storage tier for graph analytics computations performed at respective ones of a plurality of compute nodes configured for communication with the storage system over at least one network; and

switching logical connections of respective ones of the compute nodes to the at least one dispatcher node based on progress of graph analytics computations at the compute nodes;

wherein the method is performed using at least one processing device comprising a processor coupled to a memory.

18. The method of claim 17 wherein:

the graph analytics computations proceed in a series of phases each comprising at least one query;

monitoring the progress of the graph analytics computations comprises building one or more tables for data locality detection after each query ends, the one or more tables being persisted to a flash memory of the at least one dispatcher node and utilized to predict which compute nodes will become active in a next phase; and

switching logical connections of respective ones of the compute nodes comprises utilizing the predictions to logically connect different subset of the compute nodes to the at least one dispatcher node for each phase.

19. A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said processing device:

to configure a multi-tier storage system for communication over at least one network with a plurality of compute nodes, the multi-tier storage system comprising at least a front-end storage tier and a back-end storage tier;

to implement at least one dispatcher node configured to load portions of graph data from the front-end storage tier and the back-end storage tier for graph analytics computations performed at respective ones of a plurality of compute nodes configured for communication with the storage system over at least one network; and

to switch logical connections of respective ones of the compute nodes to the at least one dispatcher node based on progress of graph analytics computations at the compute nodes.

20. The non-transitory processor-readable storage medium of claim 19 wherein:

the graph analytics computations proceed in a series of phases each comprising at least one query;

monitoring the progress of the graph analytics computations comprises building one or more tables for data locality detection after each query ends, the one or more tables being persisted to a flash memory of the at least one dispatcher node and utilized to predict which compute nodes will become active in a next phase; and

switching logical connections of respective ones of the compute nodes comprises utilizing the predictions to logically connect different subset of the compute nodes to the at least one dispatcher node for each phase.