Dynamic scheduling of tasks for collecting and processing data using job configuration data
First Claim
1. A computer-implemented method, comprising:
- identifying, by a scheduler, a plurality of jobs, wherein each job in the plurality of jobs comprises collecting data from one or more sources;
transmitting, at a direction of the scheduler, credentials to each forwarder in a set of multiple forwarders, wherein each forwarder stores the credentials;
generating, by the scheduler, job configuration data for a job, wherein the job configuration data includes identification of a data collection task to perform, a data filter to apply, a target source from which the data is to be collected, and a particular indexer to receive the data after the data is collected from the target source, the particular indexer being one of a plurality of indexers configured to index data for subsequent searching; and
for each job of the plurality of jobs,selecting, by the scheduler, a particular forwarder for the job and assigning the job to the particular forwarder selected,transmitting to the particular forwarder, by the scheduler, the job configuration data that includes the identification of the data collection task to perform, the data filter to apply, the target source from which the data is to be collected, and the particular indexer to receive the data after the data is collected from the target source,using, by the particular forwarder, the job configuration data and the stored credentials to execute the job, andtransmitting, by the particular forwarder, data collected according to the job to a particular indexer of the plurality of indexers that index data for subsequent searching of the indexed data, the particular indexer being identified by the job configuration data transmitted by the scheduler to the particular forwarder.
1 Assignment
0 Petitions
Accused Products
Abstract
A scheduler manages execution of a plurality of data-collection jobs, assigns individual jobs to specific forwarders in a set of forwarders, and generates and transmits tokens (e.g., pairs of data-collection tasks and target sources) to assigned forwarders. The forwarder uses the tokens, along with stored information applicable across jobs, to collect data from the target source and forward it onto an indexer for processing. For example, the indexer can then break a data stream into discrete events, extract a timestamp from each event and index (e.g., store) the event based on the timestamp. The scheduler can monitor forwarders'"'"' job performance, such that it can use the performance to influence subsequent job assignments. Thus, data-collection jobs can be efficiently assigned to and executed by a group of forwarders, where the group can potentially be diverse and dynamic in size.
25 Citations
33 Claims
-
1. A computer-implemented method, comprising:
-
identifying, by a scheduler, a plurality of jobs, wherein each job in the plurality of jobs comprises collecting data from one or more sources; transmitting, at a direction of the scheduler, credentials to each forwarder in a set of multiple forwarders, wherein each forwarder stores the credentials; generating, by the scheduler, job configuration data for a job, wherein the job configuration data includes identification of a data collection task to perform, a data filter to apply, a target source from which the data is to be collected, and a particular indexer to receive the data after the data is collected from the target source, the particular indexer being one of a plurality of indexers configured to index data for subsequent searching; and for each job of the plurality of jobs, selecting, by the scheduler, a particular forwarder for the job and assigning the job to the particular forwarder selected, transmitting to the particular forwarder, by the scheduler, the job configuration data that includes the identification of the data collection task to perform, the data filter to apply, the target source from which the data is to be collected, and the particular indexer to receive the data after the data is collected from the target source, using, by the particular forwarder, the job configuration data and the stored credentials to execute the job, and transmitting, by the particular forwarder, data collected according to the job to a particular indexer of the plurality of indexers that index data for subsequent searching of the indexed data, the particular indexer being identified by the job configuration data transmitted by the scheduler to the particular forwarder. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)
-
-
19. A system, comprising:
-
a plurality of data processors associated with a scheduler and a set of forwarders; and a non-transitory computer-readable storage medium containing instructions which when executed on the plurality of data processors, cause the processors to perform operations including; identifying, by a scheduler, a plurality of jobs, wherein each job in the plurality of jobs comprises collecting data from one or more sources; transmitting, at a direction of the scheduler, credentials to each forwarder in a set of multiple forwarders, wherein each forwarder stores the credentials; generating, by the scheduler, job configuration data for a job, wherein the job configuration data includes identification of a data collection task to perform, a data filter to apply, a target source from which the data is to be collected, and a particular indexer in a plurality of indexers to receive the data after the data is collected from the target source; for each job of the plurality of jobs, selecting, by the scheduler, a particular forwarder for the job and assigning the job to the particular forwarder selected, transmitting to the particular forwarder, by the scheduler, the job configuration data that includes the identification of the data collection task to perform, the data filter to apply, the target source from which the data is to be collected, and the particular indexer to receive the data after the data is collected from the target source, using, by the particular forwarder, the job configuration data and the stored credentials to execute the job, and transmitting, by the particular forwarder, data collected according to the job to a particular indexer of the plurality of indexers that index data, according to the job configuration data transmitted to the particular forwarder by the scheduler, for subsequent searching of the indexed data. - View Dependent Claims (20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31)
-
-
32. A computer-program product tangibly embodied in a non- transitory machine-readable storage medium including instructions, an execution of which by one or more processors causes a plurality of data processors associated with a scheduler and a set of forwarders to:
-
identify, by a scheduler, a plurality of jobs, wherein each job in the plurality of jobs comprises collecting data from one or more sources; transmit, at a direction of the scheduler, credentials to each forwarder in a set of multiple forwarders, wherein each forwarder stores the credentials; generate, by the scheduler, job configuration data for a job, wherein the job configuration data includes identification of a data collection task to perform, a data filter to apply, a target source from which the data is to be collected, and a particular indexer in a plurality of indexers to receive the data after the data is collected from the target source; and for each job of the plurality of jobs, select, by the scheduler, a particular forwarder for the job and assign the job to the particular forwarder selected, transmit to the particular forwarder, by the scheduler, the job configuration data that includes the identification of the data collection task to perform, the data filter to apply, the target source from which the data is to be collected, and the particular indexer to receive the data after the data is collected from the target source, use, by the particular forwarder, the job configuration data and the stored credentials to execute the job, and transmit, by the particular forwarder, data collected according to the job to a particular indexer of the plurality of indexers that index data, according to the job configuration data transmitted to the particular forwarder by the scheduler, for subsequent searching of the indexed data. - View Dependent Claims (33)
-
Specification