Virtual file system providing interface between analytics platform and physical file system

• US 9,305,010 B1
• Filed: 12/30/2013
• Issued: 04/05/2016
• Est. Priority Date: 12/30/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Faibish, Sorin, Bent, John M., Tuma, George
• Primary Examiner(s): Lee, Wilson
• Application Number:
  US14/143,863
• Time in Patent Office: 827 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1847 specifically adapted to static storage, e.g. adapted to flash memory or SSD
• CPC Class Codes: G06F 16/188 Virtual file systems

Abstract

A virtual file system is arranged between a physical file system and an analytics platform comprising a plurality of compute nodes, and is configured to control movement of data between the analytics platform and the physical file system. For example, the virtual file system may be configured to present to the analytics platform a software application programming interface (API) through which the analytics platform accesses data stored in the physical file system. The analytics platform illustratively comprises a unified analytics platform that is configurable to support both database analytics and map-reduce analytics. In some embodiments, the physical file system comprises one or more of a Lustre file system, a Hadoop Distributed File System (HDFS) and a General Parallel File System (GPFS), and the virtual file system comprises a parallel log-structured file system (PLFS).

Claims

1. An apparatus comprising:

a physical file system;

a virtual file system coupled to the physical file system; and

an analytics platform comprising a plurality of compute nodes;

wherein the virtual file system is configured to control movement of data between the analytics platform and the physical file system;

wherein the analytics platform comprises a unified analytics platform operative in a first configuration to perform database analytics and operative in a second configuration to perform map-reduce analytics; and

wherein the virtual file system is implemented using at least one processing device comprising a processor coupled to a memory.

2. The apparatus of claim 1 wherein the virtual file system presents a software application programming interface to the analytics platform through which the analytics platform accesses data stored in the physical file system.

3. The apparatus of claim 1 wherein the physical file system comprises a plurality of storage tiers and the virtual file system controls movement of data between the storage tiers.

4. The apparatus of claim 3 wherein the plurality of storage tiers includes at least one tier of flash storage devices and at least one tier of disk storage devices.

5. The apparatus of claim 1 further comprising a plurality of physical file systems with the virtual file system being configured to control movement of data between the analytics platform and each of the physical file systems.

6. The apparatus of claim 5 wherein the plurality of physical file systems comprise two or more of a Lustre file system, a Hadoop Distributed File System (HDFS) and a General Parallel File System (GPFS).

7. The apparatus of claim 1 wherein the virtual file system is implemented at least in part utilizing software running on the compute nodes.

8. The apparatus of claim 1 wherein the virtual file system comprises a parallel log-structured file system (PLFS).

9. The apparatus of claim 8 wherein the PLFS comprises middleware including a plurality of synchronization daemons associated with respective storage nodes of the physical file system, and wherein a given one of the synchronization daemons is triggered by at least one of a read operation and a write operation associated with a client comprising an application running on a corresponding compute node.

10. The apparatus of claim 8 wherein the virtual file system comprises a plurality of PLFS clients running on respective ones of the compute nodes and wherein the PLFS clients are configured to control movement of data between multiple storage tiers of the physical file system in a manner that obscures said data movement from applications running on the respective compute nodes.

14. A processing platform comprising the apparatus of claim 1.

15. An information processing system comprising the apparatus of claim 1.

11. An apparatus comprising:

a physical file system;

a virtual file system coupled to the physical file system; and

an analytics platform comprising a plurality of compute nodes;

wherein the virtual file system is configured to control movement of data between the analytics platform and the physical file system;

wherein the analytics platform is configured to perform database analytics utilizing a plurality of database management system instances running on the compute nodes and the virtual file system provides a single namespace for the instances by combining database files associated with the instances into objects for storage in the physical file system utilizing the single namespace; and

wherein the virtual file system is implemented using at least one processing device comprising a processor coupled to a memory.

12. The apparatus of claim 11 wherein the objects comprise respective PLFS containers.

13. An apparatus comprising:

a physical file system;

a virtual file system coupled to the physical file system; and

an analytics platform comprising a plurality of compute nodes;

wherein the virtual file system is configured to control movement of data between the analytics platform and the physical file system;

wherein the analytics platform is configured to perform map-reduce analytics utilizing a plurality of map instances running on one or more of the compute nodes and a plurality of reduce instances running on one or more of the compute nodes and wherein the virtual file system is configured to accelerate the map-reduce analytics by controlling movement of associated data between storage tiers of the physical file system; and

wherein the virtual file system is implemented using at least one processing device comprising a processor coupled to a memory.

16. A method comprising:

configuring a virtual file system for coupling between a physical file system and an analytics platform comprising a plurality of compute nodes;

adapting the virtual file system for communication with the analytics platform;

controlling the movement of data between the analytics platform and the physical file system using the virtual file system;

configuring the analytics platform as a unified analytics platform;

operating the unified analytics platform in a first configuration to perform database analytics; and

operating the unified analytics platform in a second configuration to perform map-reduce analytics;

wherein configuring the virtual file system, adapting the virtual file system, controlling the movement of data, configuring the analytics platform and operating the unified analytics platform are implemented using at least one processing device comprising a processor coupled to a memory.

17. The method of claim 16 wherein configuring the virtual file system comprises configuring the virtual file system as a parallel log-structured file system (PLFS).

19. The method of claim 16 wherein the unified analytics platform in the first configuration performs database analytics utilizing a plurality of database management system instances running on the compute nodes and the virtual file system provides a single namespace for the instances by combining database files associated with the instances into objects for storage in the physical file system utilizing the single namespace.

20. The method of claim 16 wherein the unified analytics platform in the second configuration performs map-reduce analytics utilizing a plurality of map instances running on one or more of the compute nodes and a plurality of reduce instances running on one or more of the compute nodes and wherein the virtual file system is configured to accelerate the map-reduce analytics by controlling movement of associated data between storage tiers of the physical file system.

18. An article of manufacture comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to configure a virtual file system for coupling between a physical file system and an analytics platform comprising a plurality of compute nodes;

to adapt the virtual file system for communication with the analytics platform;

to control movement of data between the analytics platform and the physical file system using the virtual file system;

to configure the analytics platform as a unified analytics platform;

to operate the unified analytics platform in a first configuration to perform database analytics; and

to operate the unified analytics platform in a second configuration to perform map-reduce analytics.