Resource management systems and methods

• US 10,366,102 B2
• Filed: 10/20/2014
• Issued: 07/30/2019
• Est. Priority Date: 02/19/2014
• Status: Active Grant

• Current Assignee: Snowflake, Inc.
• Original Assignee: Snowflake, Inc.
• Inventors: Dageville, Benoit, Cruanes, Thierry, Zukowski, Marcin
• Primary Examiner(s): Nguyen, Dustin
• Application Number:
  US14/518,884
• Publication Number:
  US 20150237137A1
• Time in Patent Office: 1,744 Days
• Field of Search: 709201, 709223, 718 1, 718100, 718102, 718103, 707714, 707718
• CPC Class Codes: A61F 5/566 Intra-oral devices
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24545 Selectivity estimation or determination
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5016 : the resource being the memory
• CPC Class Codes: G06F 9/5044 : considering hardware capabilities
• CPC Class Codes: G06F 9/5083 : Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/5088 : involving task migration
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.

Claims

1. A system comprising:

a plurality of shared storage devices collectively storing database data comprising files;

a resource manager comprising a software program stored in memory and executed by one or more processors, the resource manager programmed to;

receive at least one database query directed to the database data by a computerized query source;

identify one or more of the files within the database data that must be processed to respond to the at least one database query;

divide processing of the one or more of the files into multiple discrete tasks; and

assign different tasks of the multiple discrete tasks to different nodes within a plurality of nodes of an execution platform;

wherein the resource manager is independent of and remote from the plurality of nodes of the execution platform;

the execution platform executing the multiple discrete tasks, wherein the execution platform comprises a plurality of virtual warehouses, and each virtual warehouse of the plurality of virtual warehouses includes at least one node of the plurality of nodes, and wherein each node of the plurality of nodes is independent of and remote from the computerized query source and from the plurality of shared storage devices and comprises;

at least one processor executing one or more tasks of the multiple discreet tasks assigned thereto by the resource manager; and

at least one local cache caching at least a portion of the database data.

2. The system of claim 1, wherein the at least one database query is in the form of one or more SQL statements.

3. The system of claim 1, wherein at least one task of the multiple discrete tasks comprises one or more reads from the plurality of shared storage devices.

4. The system of claim 1, wherein at least one task of the multiple discrete tasks comprises one or more writes to the plurality of shared storage devices.

5. The system of claim 1, wherein the resource manager is further programmed to allocate access to the plurality of shared storage devices for a predetermined time period.

6. The system of claim 1, wherein the resource manager is further programmed to identify at least one storage device of the plurality of shared storage devices that is associated with database data identified in a received SQL statement.

7. The system of claim 1, wherein the at least one local cache of each node of the plurality of nodes is configured to store database data from any device of the plurality of shared storage devices.

8. The system of claim 7, wherein each node of the plurality of nodes is connected to:

read database data from any device of the plurality of shared storage devices; and

write database data to any device of the plurality of shared storage devices.

9. The system of claim 7, wherein each node of the plurality of nodes is configured to read a needed portion of the database data from the at least one local cache corresponding thereto rather than from the plurality of shared storage devices whenever the needed portion of the database data is cached at the at least one local cache.

10. A method comprising:

deploying an interface system between a plurality of shared storage devices collectively storing database data and at least one computerized query source, wherein;

the interface system comprises a resource manager and an execution platform;

the resource manager comprises a software program stored in memory and executed by one or more processors connected to the execution platform;

the execution platform is independent of and remote from the plurality of shared storage devices;

the execution platform is independent of and remote from the at least one computerized query source;

the execution platform comprises a plurality of nodes, wherein the plurality of nodes are organized in a plurality of virtual warehouses such that each virtual warehouse of the plurality of virtual warehouses comprises multiple nodes of the plurality of nodes, each node thereof comprising at least one processor and at least one local cache caching at least a portion of the database data; and

the resource manager is independent of and remote from the plurality of nodes of the execution platform;

receiving, by the resource manager, at least one database query directed to the database data by the at least one computerized query source;

determining a plurality of tasks that must be completed to respond to the at least one database query;

distributing, by the resource manager, the plurality of tasks among the plurality of nodes for processing, wherein each node of the plurality of nodes is assigned at least one task of the plurality of tasks;

providing, by the execution platform to the resource manager, results obtained from processing of the plurality of tasks; and

using, by the resource manager, the results to respond to the at least one database query.

11. The method of claim 10, wherein the at least one database query comprises one or more SQL statements.

12. The method of claim 10, further comprising processing, by each node of the plurality of nodes, the at least one task assigned thereto by:

determining whether a portion of the database data necessary to process the at least one task is cached at the at least one local cache corresponding thereto; and

when it is determined that the portion of the database data is cached on the at least one local cache, retrieving the portion of the database data from the at least one local cache; and

when it is determined that the portion of the database data is not cached on the at least one local cache, retrieving the portion of the database data from the plurality of shared storage devices and caching the portion of the database data at the at least one local cache.

13. A system comprising:

means for storing database data;

means for receiving at least one database query directed to the database data by at least one computerized query source;

means for determining which files within the database data must be processed to respond to the at least one database query;

means for dividing processing of the files into multiple discrete tasks;

means for assigning different tasks of the multiple discrete tasks to different nodes within a plurality of nodes of an execution platform, wherein the plurality of nodes are organized as a plurality of virtual warehouses, each virtual warehouse of the plurality virtual warehouses comprises more than one node of the plurality of nodes; and

the execution platform executing the multiple discrete tasks, wherein each node of the plurality of nodes is independent of and remote from the at least one database query source and from the means for storing database data and comprises;

at least one processor executing one or more tasks of the multiple discreet tasks assigned thereto by the means for assigning; and

at least one local cache caching at least a portion of the database data, wherein the means for receiving the at least one database query is independent of and remote from the plurality of nodes of the execution platform.