Resource provisioning systems and methods

• US 10,325,032 B2
• Filed: 10/20/2014
• Issued: 06/18/2019
• Est. Priority Date: 02/19/2014
• Status: Active Grant

• Current Assignee: Snowflake, Inc.
• Original Assignee: Snowflake, Inc.
• Inventors: Dageville, Benoit, Cruanes, Thierry, Zukowski, Marcin
• Primary Examiner(s): Beausoliel, Jr., Robert W
• Assistant Examiner(s): Ganger, Arthur
• Application Number:
  US14/518,826
• Publication Number:
  US 20150234902A1
• Time in Patent Office: 1,702 Days
• CPC Class Codes: A61F 5/566 Intra-oral devices
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/24545 Selectivity estimation or determination
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5016 : the resource being the memory
• CPC Class Codes: G06F 9/5044 : considering hardware capabilities
• CPC Class Codes: G06F 9/5083 : Techniques for rebalancing the load in a distributed system
• CPC Class Codes: G06F 9/5088 : involving task migration
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Example resource provisioning systems and methods are described. In one implementation, an execution platform accesses multiple remote storage devices. The execution platform includes multiple virtual warehouses, each of which includes a cache to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices. A resource manager is coupled to the execution platform and monitors received data processing requests and resource utilization. The resource manager also determines whether additional virtual warehouses are needed based on the data processing requests and the resource utilization. If additional virtual warehouses are needed, the resource manager provisions a new virtual warehouse.

Claims

1. A system comprising:

an execution platform configured to access a plurality of remote storage devices, the plurality of remote storage devices collectively storing database data, the execution platform including a plurality of virtual warehouses that each share the plurality of remote storage devices, retrieve data from the plurality of remote storage devices, and execute one or more queries received from client devices against the database data, each virtual warehouse including a cache configured to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices; and

a resource manager coupled to the execution platform and configured to;

determine predicted data processing requests based on previous patterns of previously received data processing requests from a particular client at a particular time;

monitor data processing requests including received data processing requests and the predicted data processing requests;

monitor resource utilization in the plurality of virtual warehouses;

determine whether additional virtual warehouses are needed based on the data processing requests and the resource utilization in the plurality of virtual warehouses; and

responsive to determining that an additional virtual warehouse is needed, provision a new virtual warehouse without increasing an amount of storage in the plurality of remote storage devices.

2. The system of claim 1, wherein the resource manager is further configured to monitor query response rates.

3. The system of claim 1, wherein the resource manager is further configured to monitor current resource utilization associated with the plurality of virtual warehouses, and wherein determining whether additional virtual warehouses are needed is further based on the current resource utilization.

4. The system of claim 1, wherein the resource manager is further configured to monitor at least one of a number of queries running concurrently, a percentage of maximum load, and whether any query processing is delayed.

5. The system of claim 1, wherein the resource manager is further configured to deactivate the new virtual warehouse responsive to determining that the new virtual warehouse is no longer needed.

6. The system of claim 1, wherein the resource manager is further configured to:

receive a notice of an upcoming data processing project; and

responsive to receiving the notice of the upcoming data processing project, create a new virtual warehouse associated with the upcoming data processing project.

7. The system of claim 6, wherein the resource manager is further configured to deactivate the new virtual warehouse upon completion of the data processing project.

8. The system of claim 6, wherein the upcoming data processing project has a predetermined time duration, and wherein the new virtual warehouse is deactivated after the predetermined time duration.

9. The system of claim 1, wherein the resource manager is further configured to:

receive a request to perform a plurality of data processing activities; and

responsive to receiving the request to perform the plurality of data processing activities, automatically create a new virtual warehouse associated with the received request.

10. A method comprising:

determining predicted data processing requests based on previous patterns of previously received data processing requests from a particular client at a particular time;

monitoring data processing requests including received data processing requests and the predicted data processing requests, wherein the data processing requests are associated with data stored in a plurality of remote storage devices collectively storing database data;

monitoring resource utilization in a plurality of virtual warehouses, wherein each virtual warehouse includes a cache configured to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices, wherein the plurality of virtual warehouses each share the plurality of remote storage devices, retrieve data from the plurality of remote storage devices, and execute queries against the database data;

determining whether additional virtual warehouses are needed based on the data processing requests and the resource utilization in the plurality of virtual warehouses; and

creating a new virtual warehouse responsive to determining that an additional virtual warehouse is needed without increasing an amount of storage in the plurality of remote storage devices.

11. The method of claim 10, further comprising monitoring query response rates, wherein determining whether additional virtual warehouses are needed is further based on the query response rates.

12. The method of claim 10, further comprising monitoring a number of users associated with each of the plurality of virtual warehouses, wherein determining whether additional virtual warehouses are needed is further based on the number of users associated with each of the plurality of virtual warehouses.

13. The method of claim 10, further comprising:

determining whether the new virtual warehouse is no longer needed; and

deactivating the new virtual warehouse responsive to determining that the new virtual warehouse is no longer needed.

14. The method of claim 10, further comprising monitoring a number of queries running concurrently, wherein determining whether additional virtual warehouses are needed is further based on the number of queries running concurrently.

15. The method of claim 10, further comprising determining whether execution of any queries are currently delayed, wherein determining whether additional virtual warehouses are needed is further based on whether any queries are currently delayed.

16. The method of claim 10, further comprising:

receiving a notice of an upcoming data processing project;

creating a new virtual warehouse associated with the upcoming data processing project; and

deactivating the new virtual warehouse upon completion of the data processing project.

17. The method of claim 10, further comprising:

receiving a notice of an upcoming data processing project, wherein the upcoming data processing project has a predetermined time period;

creating a new virtual warehouse associated with the upcoming data processing project; and

deactivating the new virtual warehouse after the predetermined time period.

18. A system comprising:

means for accessing a plurality of remote storage devices collectively storing database data, the means for accessing includes a plurality of virtual warehouses that each share the plurality of remote storage devices, retrieve data from the plurality of remote storage devices, and execute one or more queries received from client devices against the database data, each virtual warehouse including a cache configured to store data retrieved from the remote storage devices and a processor that is independent of the remote storage devices; and

means for managing resources coupled to the means for accessing and configured to;

determine predicted data processing requests based on previous patterns of previously received data processing requests from a particular client at a particular time;

monitor data processing requests including received data processing requests and the predicted data processing requests;

monitor resource utilization in the plurality of virtual warehouses;

determine whether additional virtual warehouses are needed based on the data processing requests and the resource utilization in the plurality of virtual warehouses; and

responsive to determining that an additional virtual warehouse is needed, provision a new virtual warehouse that share the plurality of remote storage devices and execute one or more queries received from client devices against the database data.

19. The system of claim 18, wherein the means for managing resources is further configured to deactivate the new virtual warehouse upon determination that the new virtual warehouse is no longer needed.

20. The system of claim 18, wherein the means for managing resources is further configured to:

predict a time required to execute a query; and

provision a new virtual warehouse if the time required to execute the query exceeds a threshold value.