Tiering with pluggable storage system for parallel query engines

• US 10,719,510 B2
• Filed: 11/16/2017
• Issued: 07/21/2020
• Est. Priority Date: 02/25/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Tiwari, Sameer, Bhandarkar, Milind Arun, Mogal, Bhooshan Deepak
• Primary Examiner(s): Lewis, Cheryl
• Application Number:
  US15/814,695
• Publication Number:
  US 20180075052A1
• Time in Patent Office: 978 Days
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24524 Access plan code generation and invalidation; Reuse of access plans
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H05K 999/99 dummy group

Abstract

A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes determining a usage level of a file, wherein the file is stored in a first storage system, moving the file to a second storage system based on the determined usage level of the file updating location information in a catalog based on the movement of the file, and performing at least a portion of a query on the file after updating location information in the catalog.

Claims

1. A computer implemented method, comprising:

receiving, by a universal node comprising one or more processors, a query from a client;

accessing, by the universal node, a catalog comprising location information for one or more files stored across one or more storage systems, the one or more storage systems comprising a first storage system and a second storage system, and the universal node being configured to interface with the first storage system and the second storage system; and

providing, by the universal node, a universal namespace to the client, wherein the universal namespace collectively presents information for the one or more files stored in a plurality of namespaces corresponding to the one or more storage systems, wherein a files of the one or more files is accessible via the universal namespace such that the client accesses the file at a same address of the universal namespace regardless of whether the file is moved from the first storage system to the second storage system.

2. The method of claim 1, wherein a catalog service updates a catalog with a location of the file based on a storage system of the plurality of storage systems on which the file is stored.

3. The method of claim 1, further comprising:

determining, by one or more processors, a usage level of the file, wherein the file is stored in the first storage system; and

moving, by one or more processors, the file to the second storage system based on the determined usage level of the file.

4. The method of claim 3, further comprising:

updating, by one or more processors, location information of the catalog based on the movement of the file.

5. The method of claim 3, further comprising:

performing at least a portion of the query on the file after updating location information in the catalog, wherein the universal node comprises a universal job tracker that tracks a status of a one or more jobs corresponding to the query.

6. The method of claim 5, wherein the universal job tracker obtains a corresponding status of at least a portion of the query from one or more data node job trackers.

7. The method of claim 3, wherein moving the file to the second storage system based on the determined usage level includes moving the file to a high performance storage system based on high usage level.

8. The method of claim 7, wherein the second storage system is a higher performance storage system than the first storage system.

9. The method of claim 3, wherein the usage level includes a usage frequency.

10. The method of claim 9, further comprising determining a predicted usage time based on the usage frequency.

11. The method of claim 10, wherein moving the file to the second storage system based on the determined usage level includes moving the file to the second storage system before the predicted usage time.

12. The method of claim 11, further comprising moving the file to the first storage system after the predicted usage time.

13. The method of claim 1, wherein in event that the client is to access the file, the client accesses the file via the universal namespace using the corresponding address of the file in relation to the universal namespace.

14. The method of claim 13, wherein the client does not directly use an address of the file in relation to the particular storage system of the plurality of storage system on which the file is located.

15. The method of claim 1, wherein the universal node comprises a plurality of protocols for a plurality of file systems, and wherein the universal node uses corresponding ones of the plurality of protocols in connection with accessing the first storage system and the second storage system.

16. The method of claim 1, wherein in the event that the file is moved from the first storage system to the second storage system, the file is accessible via the universal namespace and the move of the file from the first storage system to the second storage system is transparent to the client.

17. The method of claim 1, wherein the first storage system and the second storage system use different file systems.

18. The method of claim 1, further comprising:

in response to receiving the query, determining a location of the one or more files in the universal namespace using the catalog service; and

in the event that the one or more files are located at the first storage system, sending a job corresponding to at least a portion of the query to the first storage system.

19. The method of claim 1, wherein the location information in the catalog is updated by a catalog service.

20. The method of claim 1, wherein the catalog stores location information indicating a location for a plurality of files located at a plurality storage systems corresponding to a plurality of namespaces.

21. The method of claim 1, wherein the file is accessible via the universal namespace such that the client accesses the file at a same address of the universal namespace regardless of whether the file is moved from the first storage system to the second storage system.

22. A system, comprising:

one or more processors configured to;

receive a query from a client;

access a catalog comprising location information for one or more files stored across one or more storage systems, the one or more storage systems comprising a first storage system and a second storage system, and a universal node being configured to interface with the first storage system and the second storage system; and

provide a universal namespace to the client, wherein the universal namespace collectively presents information for the one or more files stored in a plurality of namespaces corresponding to the one or more storage systems, wherein a files of the one or more files is accessible via the universal namespace such that the client accesses the file at a same address of the universal namespace regardless of whether the file is moved from the first storage system to the second storage system; and

a memory coupled to the processor and configured to provide the processor with instructions.

23. A computer program product for processing data, comprising a non-transitory computer readable medium having program instructions implemented by a processor circuitry when executed, the program instructions comprising instructions for:

receiving, by a universal node comprising one or more processors, a query from a client;

accessing, by the universal node, a catalog comprising location information for one or more files stored across one or more storage systems, the one or more storage systems comprising a first storage system and a second storage system, and the universal node being configured to interface with the first storage system and the second storage system; and

providing, by the universal node, a universal namespace to the client, wherein the universal namespace collectively presents information for the one or more files stored in a plurality of namespaces corresponding to the one or more storage systems, wherein a files of the one or more files is accessible via the universal namespace such that the client accesses the file at a same address of the universal namespace regardless of whether the file is moved from the first storage system to the second storage system.