Query processing using query-resource usage and node utilization data
First Claim
1. A method comprising:
- tracking query-resource usage data corresponding to an indication of resources used to execute each of a plurality of previous queries and timing data corresponding to an execution time to obtain results for at least some of the plurality of previous queries;
tracking node resource utilization data related to a plurality of nodes, each node comprising one or more processors;
receiving, at a data intake and query system, a query identifying a set of data located in one or more data sources to be processed and a manner of processing the set of data;
defining, by the data intake and query system, a query processing scheme for obtaining and processing the set of data, wherein the query processing scheme specifies an amount of node resources to allocate to the query based at least in part on available node resources determined using the node resource utilization data and a comparison of the query with one or more of the plurality of previous queries associated with the query-resource usage data; and
executing the query based on the query processing scheme.
1 Assignment
0 Petitions
Accused Products
Abstract
Systems and methods are disclosed for processing queries against one or more dataset sources. The system tracks query resource data and resource utilization data. The query-resource usage data can indicate resources used to execute queries. The node resource utilization data can indicate current utilization of nodes in the system. Upon receipt of a query that identifies a set of data to be processed and a manner of processing the set of data, the system can use the query-resource usage data and the resource utilization data to define a query processing scheme. The query can then be executed using the query processing scheme. In some cases, the query coordinator can dynamically allocate partitions operating on worker nodes to execute the query.
-
Citations
30 Claims
-
1. A method comprising:
-
tracking query-resource usage data corresponding to an indication of resources used to execute each of a plurality of previous queries and timing data corresponding to an execution time to obtain results for at least some of the plurality of previous queries; tracking node resource utilization data related to a plurality of nodes, each node comprising one or more processors; receiving, at a data intake and query system, a query identifying a set of data located in one or more data sources to be processed and a manner of processing the set of data; defining, by the data intake and query system, a query processing scheme for obtaining and processing the set of data, wherein the query processing scheme specifies an amount of node resources to allocate to the query based at least in part on available node resources determined using the node resource utilization data and a comparison of the query with one or more of the plurality of previous queries associated with the query-resource usage data; and executing the query based on the query processing scheme. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26)
-
-
27. A computing system, comprising:
one or more processing devices configured to; track query-resource usage data corresponding to an indication of resources used to execute each of a plurality of previous queries and timing data corresponding to an execution time to obtain results for at least some of the plurality of previous queries; track node resource utilization data related to a plurality of nodes, each node comprising one or more processors; receive a query identifying a set of data located in one or more data sources to be processed and a manner of processing the set of data; define a query processing scheme for obtaining and processing the set of data, wherein the query processing scheme specifies an amount of node resources to allocate to the query based at least in part on available node resources determined using the node resource utilization data and a comparison of the query with one or more of the plurality of previous queries associated with the query-resource usage data; and execute the query based on the query processing scheme. - View Dependent Claims (28, 29)
-
30. Non-transitory computer readable media comprising computer-executable instructions that, when executed by a computing system, cause the computing system to:
-
track query-resource usage data corresponding to an indication of resources used to execute each of a plurality of previous queries and timing data corresponding to an execution time to obtain results for at least some of the plurality of previous queries; track node resource utilization data related to a plurality of nodes, each node comprising one or more processors; receive a query identifying a set of data located in one or more data sources to be processed and a manner of processing the set of data; define a query processing scheme for obtaining and processing the set of data, wherein the query processing scheme specifies an amount of node resources to allocate to the query based at least in part on available node resources determined using the node resource utilization data and a comparison of the query with one or more of the plurality of previous queries associated with the query-resource usage data; and execute the query based on the query processing scheme.
-
Specification