Pluggable storage system for parallel query engines

• US 9,805,053 B1
• Filed: 03/15/2013
• Issued: 10/31/2017
• Est. Priority Date: 02/25/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Tiwari, Sameer, Bhandarkar, Milind Arun, Mogal, Bhooshan Deepak
• Primary Examiner(s): Coby, Frantz
• Application Number:
  US13/842,816
• Time in Patent Office: 1,691 Days
• Field of Search: 707802, 707812, 707821, 707776, 707661, 709201, 709203, 709204, 709213, 709219
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24524 Access plan code generation and invalidation; Reuse of access plans
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H05K 999/99 dummy group

Abstract

A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes, receiving a query from a client, based on the received query, analyzing a catalog for location information, based on the analysis, determining a first storage system, an associated first file system and an associated first protocol, using the associated first protocol to communicate with the first storage system, and performing at least a portion of the query on the first storage system.

Claims

1. A method for managing data, comprising:

receiving, by a universal namenode, a query from a client;

based at least in part on the received query, accessing a catalog service and searching a catalog provided by the catalog service for location information of one or more files responsive to the query, wherein the catalog stores a mapping of a plurality of files stored on a plurality of storage systems to a location at which the plurality of files are respectively stored on the corresponding plurality of storage systems, the plurality of storage systems comprising at least a first storage system and a second storage system;

based at least in part on the search of the catalog, determining to move at least one of the plurality of files from the second storage system to the first storage system, and determining to communicate with the first storage system in connection with the one or more files responsive to the queries, and a first protocol for communication with the first storage system;

communicating, by the universal namenode, with the first storage system using the associated first protocol;

performing at least a portion of the query on the first storage system; and

providing, to the client, results of the query such that in the event that various portions of the results correspond to query results stored on a set of the plurality of storage systems, the results of the query are presented in a unified view across the set of the plurality of storage systems and appear, from a perspective of the client, to exist from a single namespace.

2. The method as recited in claim 1, further comprising the first storage system reporting back the results of at least a portion of the query.

3. The method as recited in claim 1, further comprising adding a new storage system.

4. The method as recited in claim 3, further comprising determining a new protocol for communication with the new storage system.

5. The method as recited in claim 4, wherein a new file system associated with the new storage system is the same as a first file system associated with the first storage system.

6. The method as recited in claim 4, wherein a new file system associated with the new storage system is not the same as a first file system associated with the first storage system.

7. The method as recited in claim 4, further comprising updating the catalog based on data stored in the new storage system.

8. The method as recited in claim 1, further comprising determining a second storage system, and a second protocol for communication with the second storage system.

21. The method of claim 1, wherein the first protocol for communication with the first storage system is determined, based at least in part, on a first file system associated with the first storage system.

22. The method of claim 1, further comprising:

determining whether to move a first file that is stored on the first storage system to another storage system based at least in part on a frequency with which the first file is accessed;

in the event that the first file is determined to be moved from the first storage system to the other storage system, moving the first file to the other storage system; and

updating location information associated with the first file stored in the catalog to point from a location at which the first file was stored in the first storage system to a location at which the first file is stored in the other storage system, wherein, from the perspective of the client, the a location of the first file is the same in relation to the singe namespace regardless of whether the first file is moved from the first storage system to the other storage system.

23. The method of claim 22, wherein the updating of the location information comprises updating the mapping of the plurality of files stored on the plurality of storage systems to the location at which the plurality of files so as to update a record corresponding to the first file.

24. The method of claim 1, wherein the query received from the client is a query to identify one or more files stored across a distributed storage system comprising the plurality of storage systems.

25. The method of claim 1, further comprising:

tracking, by a universal job tracker of the universal namenode, a status of one or more portions of the query being respectively performed by corresponding ones of the plurality of storage systems.

26. The method of claim 1, wherein the determining to move the at least one of the plurality of files from the second storage system to the first storage system based at least in part on the search of the catalog comprises:

determining to move the at least one of the plurality of files in response to the receiving the query from the client.

9. A system for managing data, comprising a processor configured to:

receive a query from a client;

based at least in part on the received query, accessing a catalog service and searching a catalog provided by the catalog service for location information of one or more files responsive to the query, wherein the catalog stores a mapping of a plurality of files stored on a plurality of storage systems to a location at which the plurality of files are respectively stored on the corresponding plurality of storage systems, the plurality of storage systems comprising at least a first storage system and a second storage system;

based at least in part on the search of the catalog, determine to move at least one of the plurality of files from the second storage system to the first storage system, and determine to communicate with the first storage system in connection with the one or more files responsive to the queries, and a first protocol for communication with the first storage system;

communicate with the first storage system using the associated first protocol;

perform at least a portion of the query on the first storage system; and

provide, to the client, results of the query such that in the event that various portions of the results correspond to query results stored on a set of the plurality of storage systems, the results of the query are presented in a unified view across the set of the plurality of storage systems and appear, from a perspective of the client, to exist from a single namespace.

10. The system as recited in claim 9, the processor further configured to add a new storage system.

11. The system as recited in claim 10, the processor further configured to determine a new protocol for communication with the new storage system.

12. The system as recited in claim 11, wherein a new file system associated with the new storage system is the same as a first file system associated with the first storage system.

13. The system as recited in claim 11, wherein a new file system associated with the new storage system is not the same as a first file system associated with the first storage system.

20. The method of claim 11, wherein the new protocol is not the same as the first protocol.

14. A computer program product for processing data, comprising a non-transitory computer readable medium having program instructions embodied therein for:

receiving a query from a client;

based at least in part on the received query, accessing a catalog service and searching a catalog provided by the catalog service for location information of one or more files responsive to the query, wherein the catalog stores a mapping of a plurality of files stored on a plurality of storage systems to a location at which the plurality of files are respectively stored on the corresponding plurality of storage systems, the plurality of storage systems comprising at least a first storage system and a second storage system;

based at least in part on the search of the catalog, determining to move at least one of the plurality of files from the second storage system to the first storage system, and determining to communicate with the first storage system in connection with the one or more files responsive to the queries, a first protocol for communication with the first storage system;

communicate with the first storage system using the associated first protocol;

performing at least a portion of the query on the first storage system; and

providing, to the client, results of the query such that in the event that various portions of the results correspond to query results stored on a set of the plurality of storage systems, the results of the query are presented in a unified view across the set of the plurality of storage systems and appear, from a perspective of the client, to exist from a single namespace.

15. The computer program product as recited in claim 14, further comprising program instructions for the first storage system reporting back the results of at least a portion of the query.

16. The computer program product as recited in claim 14, further comprising program instructions for adding a new storage system.

17. The computer program product as recited in claim 16, further comprising program instructions for determining a new protocol for communication with the new storage system.

18. The computer program product as recited in claim 17, wherein a new file system associated with the new storage system is the same as a first file system associated with the first storage system.

19. The computer program product as recited in claim 17, wherein a new file system is not the same as a first file system associated with the first storage system.