Resource provisioning systems and methods

• US 9,576,039 B2
• Filed: 10/20/2014
• Issued: 02/21/2017
• Est. Priority Date: 02/19/2014
• Status: Active Grant

• Current Assignee: Snowflake, Inc.
• Original Assignee: Snowflake Computing Inc. (Snowflake, Inc.)
• Inventors: Dageville, Benoit, Cruanes, Thierry, Zukowski, Marcin
• Primary Examiner(s): DAO, THUY CHAN
• Application Number:
  US14/518,898
• Publication Number:
  US 20150234682A1
• Time in Patent Office: 855 Days
• US Class Current: 1/1
• CPC Class Codes: A61F 5/566 Intra-oral devices
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24545 Selectivity estimation or determination
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5016 : the resource being the memory
• CPC Class Codes: G06F 9/5044 : considering hardware capabilities
• CPC Class Codes: G06F 9/5083 : Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/5088 : involving task migration
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Example resource provisioning systems and methods are described. In one implementation, an execution platform accesses multiple remote storage devices. The execution platform includes multiple virtual warehouses, each of which includes a cache to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices. A resource manager is coupled to the execution platform and monitors received data processing requests and resource utilization. The resource manager also determines whether additional virtual warehouses are needed based on the data processing requests and the resource utilization. If additional virtual warehouses are needed, the resource manager provisions a new virtual warehouse.

Claims

1. A method comprising:

controlling, by a resource manager, an execution platform comprising a plurality of nodes, each node thereof comprising (1) at least one processor and (2) at least one local cache, wherein the resource manager comprises a software program stored in memory and executed by one or more processors connected to the execution platform;

executing, by the execution platform, data processing requests that originate from a plurality of computerized query sources independent of the execution platform and are directed to data collectively stored in a plurality of shared storage devices independent of the execution platform;

caching, by the execution platform, at least a portion of the data within the at least one local cache of each node of the plurality of nodes;

monitoring, by the resource manager, the data processing requests that originate from the plurality of query sources and are directed to the data;

determining, by the resource manager, that at least one of additional data storage capacity and additional processing capacity are needed based on the data processing requests; and

increasing, by the resource manager, in response to the determining, an amount of the data cached by the execution platform by adding one or more nodes to the plurality of nodes, wherein each of the one or more nodes added comprise (1) at least one processor and (2) at least one local cache.

2. The method of claim 1, further comprising monitoring, by the resource manager, query response rates corresponding to the plurality of query sources, wherein the determining that at least one of additional data storage capacity and additional processing capacity are needed is further based on the query response rates.

3. The method of claim 1, further comprising concluding, by the resource manager based on the data processing requests, that at least one of the additional data storage capacity and the additional processing capacity are no longer needed.

4. The method of claim 1, further comprising decreasing, by the resource manager in response to the concluding, the amount of the data cached by the execution platform by removing at least one node from the plurality of nodes.

5. The method of claim 1, wherein the determining comprises:

determining, by the resource manager, current query processing delays; and

determining, by the resource manager, that the current query processing delays exceed a threshold.

6. A method for storing and retrieving data, the method comprising:

deploying an execution platform as an interface between (1) a plurality of remote, shared storage devices collectively storing data and (2) a plurality of computerized query sources sending queries directed to the data, wherein

the execution platform is independent of the plurality of remote, shared storage devices,

the execution platform is independent of the plurality of computerized query sources, and

the execution platform comprises a plurality of nodes, each node thereof comprising (1) at least one processor and (2) at least one local cache caching at least a portion of the data;

monitoring, by a resource manager, current processing delays of the execution platform as the execution platform processes queries directed to the data by the plurality of computerized query sources, wherein the resource manager comprises a software program stored in memory and executed by one or more processors connected to the execution platform;

determining, by the resource manager, that the current processing delays are greater than a threshold; and

increasing, by the resource manager in response to the determining, an amount of the data cached by the execution platform by adding one or more nodes to the plurality of nodes, wherein each of the one or more nodes added comprise (1) at least one processor and (2) at least one local cache.

7. An apparatus comprising:

a plurality of shared storage devices collectively storing data;

an execution platform independent of the plurality of shared storage devices, the execution platform executing data processing requests directed to the data by a plurality of computerized query sources;

the execution platform comprising a plurality of nodes, each node thereof being independent of the plurality of computerized query sources and comprising (1) at least one processor and (2) at least one local cache caching at least a portion of the data; and

a resource manager comprising a software program stored in memory and executed by one or more processors connected to the execution platform, the resource manager programmed to;

monitor received data processing requests directed to the data;

determine that at least one of additional data storage capacity and additional processing capacity are needed based on the data processing requests; and

increase, in response to the determining, an amount of the data cached by the execution platform by adding one or more nodes to the plurality of nodes, wherein each of the one or more nodes added comprise (1) at least one processor and (2) at least one local cache.

8. The apparatus of claim 7, wherein the resource manager is further programmed to monitor query response rates corresponding to the data, and wherein the determining that at least one of additional data storage capacity and additional processing capacity are needed is further based on the query response rates.

9. The apparatus of claim 7, wherein the resource manager is further programmed to conclude, based on the data processing requests, that at least one of the additional data storage capacity and the additional processing capacity are no longer needed.

10. The apparatus of claim 9, wherein the resource manager is further programmed to decrease, in response to the concluding, the amount of the data cached by the execution platform by removing at least one node from the plurality of nodes.

11. The apparatus of claim 7, wherein the plurality of nodes are arranged in a plurality of virtual warehouses.

12. The apparatus of claim 11, wherein the plurality of virtual warehouses are grouped into a plurality of virtual warehouse groups, and wherein each of the plurality of virtual warehouse groups is associated with a different group of users.

13. The apparatus of claim 11, wherein the plurality of virtual warehouses are grouped into a plurality of virtual warehouse groups, and wherein each of the plurality of virtual warehouse groups is associated with a different entity.