RESOURCE MANAGEMENT SYSTEMS AND METHODS
First Claim
Patent Images
1. An apparatus comprising:
- a resource manager configured to manage data processing tasks associated with a plurality of data elements;
an execution platform coupled to the resource manager, the execution platform including;
a plurality of execution nodes configured to store data retrieved from a plurality of remote storage devices, each of the plurality of execution nodes including;
a cache; and
a processor, wherein the cache and the processor are independent of the remote storage devices;
a metadata manager configured to access metadata associated with at least a portion of the plurality of data elements.
2 Assignments
0 Petitions
Accused Products
Abstract
Example resource management systems and methods are described. In one implementation, a resource manager is configured to manage data processing tasks associated with multiple data elements. An execution platform is coupled to the resource manager and includes multiple execution nodes configured to store data retrieved from multiple remote storage devices. Each execution node includes a cache and a processor, where the cache and processor are independent of the remote storage devices. A metadata manager is configured to access metadata associated with at least a portion of the multiple data elements.
-
Citations
20 Claims
-
1. An apparatus comprising:
-
a resource manager configured to manage data processing tasks associated with a plurality of data elements; an execution platform coupled to the resource manager, the execution platform including; a plurality of execution nodes configured to store data retrieved from a plurality of remote storage devices, each of the plurality of execution nodes including; a cache; and a processor, wherein the cache and the processor are independent of the remote storage devices; a metadata manager configured to access metadata associated with at least a portion of the plurality of data elements.
-
-
2. The apparatus of claim 1, wherein the plurality of data processing tasks are associated with a plurality of SQL statements.
-
3. The apparatus of claim 1, wherein the plurality of data processing tasks include tasks that read data from the plurality of remote data storage devices.
-
4. The apparatus of claim 1, wherein the plurality of data processing tasks include tasks that write data to the plurality of remote storage devices.
-
5. The apparatus of claim 1, wherein the resource management system is further configured to allocate access to the plurality of remote storage devices for a predetermined time period.
-
6. The apparatus of claim 1, wherein the resource manager is further configured to identify at least one remote storage device associated with data identified in a received SQL statement.
-
7. The apparatus of claim 1, wherein the execution platform is further configured to implement a plurality of virtual warehouses.
-
8. The apparatus of claim 7, wherein each virtual warehouse includes at least one of the plurality of execution nodes.
-
9. The apparatus of claim 1, wherein the cache is configured to store data from any of the remote storage devices.
-
10. The apparatus of claim 8, wherein each of the plurality of execution nodes is configured to:
-
read data from any of the remote storage devices; and write data to any of the remote storage devices.
-
-
11. The apparatus of claim 8, wherein each execution node is configured to read data from the cache in the execution node instead of from the remote storage devices.
-
12. The apparatus of claim 7, wherein the execution platform is configured to create new virtual warehouses responsive to instructions from the resource manager.
-
13. The apparatus of claim 1, wherein the resource manager is further configured to receive a plurality of data requests.
-
14. A method comprising:
-
receiving a data processing request, wherein the data processing request is associated with a plurality of remote storage devices storing a plurality of data elements; determining a plurality of tasks necessary to process the received data processing request; distributing the plurality of tasks to a plurality of execution nodes for processing, wherein each of the plurality of execution nodes includes a cache configured to store data retrieved from the remote storage devices, and wherein each of the plurality of execution nodes uses data contained in the cache to process one of a plurality of tasks, including; receiving task results from the plurality of execution nodes; and generating a statement result based on the task results.
-
-
15. The method of claim 14, further comprising creating a query coordinator to manage processing of the received data processing request.
-
16. The method of claim 14, wherein the plurality of execution nodes are segmented into a plurality of virtual warehouses.
-
17. The method of claim 14, each of the plurality of execution nodes:
-
determining whether data necessary to process an assigned task is contained in the execution node'"'"'s cache; and responsive to determining that the data necessary to process the assigned task is not contained in the execution node'"'"'s cache; retrieving data from the remote storage devices; and storing the retrieved data in the execution node'"'"'s cache.
-
-
18. An apparatus comprising:
-
means for managing data processing tasks associated with a plurality of data elements in a plurality of remote data storage devices; means for executing data processing tasks coupled to the means for managing data processing tasks, the means for executing data processing tasks including; a plurality of caches configured to store data retrieved from the remote storage devices; and a plurality of processors that are independent of the remote storage devices; and means for managing metadata configured to access metadata associated with at least a portion of the plurality of data elements.
-
-
19. The apparatus of claim 18, wherein the means for executing data processing tasks is implemented as a plurality of virtual warehouses.
-
20. The apparatus of claim 19, wherein each virtual warehouse includes a plurality of execution nodes, and each execution node includes:
-
a cache; and a processor.
-
Specification