RESOURCE MANAGEMENT SYSTEMS AND METHODS

• US 20150237137A1
• Filed: 10/20/2014
• Published: 08/20/2015
• Est. Priority Date: 02/19/2014
• Status: Active Grant

• Current Assignee: Snowflake, Inc.
• Original Assignee: Snowflake Computing Inc. (Snowflake, Inc.)
• Inventors: Dageville, Benoit, Cruanes, Thierry, Zukowski, Marcin
• Granted Patent: US 10,366,102 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A61F 5/566 Intra-oral devices
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24545 Selectivity estimation or determination
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5016 : the resource being the memory
• CPC Class Codes: G06F 9/5044 : considering hardware capabilities
• CPC Class Codes: G06F 9/5083 : Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/5088 : involving task migration
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.

Claims

1. An apparatus comprising:

a resource manager configured to manage data processing tasks associated with a plurality of data elements;

an execution platform coupled to the resource manager, the execution platform including;

a plurality of execution nodes configured to store data retrieved from a plurality of remote storage devices, each of the plurality of execution nodes including;

a cache; and

a processor, wherein the cache and the processor are independent of the remote storage devices;

a metadata manager configured to access metadata associated with at least a portion of the plurality of data elements.

2. The apparatus of claim 1, wherein the plurality of data processing tasks are associated with a plurality of SQL statements.

3. The apparatus of claim 1, wherein the plurality of data processing tasks include tasks that read data from the plurality of remote data storage devices.

4. The apparatus of claim 1, wherein the plurality of data processing tasks include tasks that write data to the plurality of remote storage devices.

5. The apparatus of claim 1, wherein the resource management system is further configured to allocate access to the plurality of remote storage devices for a predetermined time period.

6. The apparatus of claim 1, wherein the resource manager is further configured to identify at least one remote storage device associated with data identified in a received SQL statement.

7. The apparatus of claim 1, wherein the execution platform is further configured to implement a plurality of virtual warehouses.

8. The apparatus of claim 7, wherein each virtual warehouse includes at least one of the plurality of execution nodes.

9. The apparatus of claim 1, wherein the cache is configured to store data from any of the remote storage devices.

10. The apparatus of claim 8, wherein each of the plurality of execution nodes is configured to:

read data from any of the remote storage devices; and

write data to any of the remote storage devices.

11. The apparatus of claim 8, wherein each execution node is configured to read data from the cache in the execution node instead of from the remote storage devices.

12. The apparatus of claim 7, wherein the execution platform is configured to create new virtual warehouses responsive to instructions from the resource manager.

13. The apparatus of claim 1, wherein the resource manager is further configured to receive a plurality of data requests.

14. A method comprising:

receiving a data processing request, wherein the data processing request is associated with a plurality of remote storage devices storing a plurality of data elements;

determining a plurality of tasks necessary to process the received data processing request;

distributing the plurality of tasks to a plurality of execution nodes for processing, wherein each of the plurality of execution nodes includes a cache configured to store data retrieved from the remote storage devices, and wherein each of the plurality of execution nodes uses data contained in the cache to process one of a plurality of tasks, including;

receiving task results from the plurality of execution nodes; and

generating a statement result based on the task results.

15. The method of claim 14, further comprising creating a query coordinator to manage processing of the received data processing request.

16. The method of claim 14, wherein the plurality of execution nodes are segmented into a plurality of virtual warehouses.

17. The method of claim 14, each of the plurality of execution nodes:

determining whether data necessary to process an assigned task is contained in the execution node'"'"'s cache; and

responsive to determining that the data necessary to process the assigned task is not contained in the execution node'"'"'s cache;

retrieving data from the remote storage devices; and

storing the retrieved data in the execution node'"'"'s cache.

18. An apparatus comprising:

means for managing data processing tasks associated with a plurality of data elements in a plurality of remote data storage devices;

means for executing data processing tasks coupled to the means for managing data processing tasks, the means for executing data processing tasks including;

a plurality of caches configured to store data retrieved from the remote storage devices; and

a plurality of processors that are independent of the remote storage devices; and

means for managing metadata configured to access metadata associated with at least a portion of the plurality of data elements.

19. The apparatus of claim 18, wherein the means for executing data processing tasks is implemented as a plurality of virtual warehouses.

20. The apparatus of claim 19, wherein each virtual warehouse includes a plurality of execution nodes, and each execution node includes:

a cache; and

a processor.