Scheduling jobs for processing log files using a database system

• US 10,275,281 B2
• Filed: 09/30/2016
• Issued: 04/30/2019
• Est. Priority Date: 09/30/2016
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Pradeep, Aakash, Sreenivasa, Abhishek, Torman, Adam, Warshavsky, Alex, Jain, Samarpan, Bandyopadhyay, Soumen
• Primary Examiner(s): Kim, Dong U
• Application Number:
  US15/281,559
• Publication Number:
  US 20180095790A1
• Time in Patent Office: 942 Days
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/1479 Generic software techniques for error detection or fault masking
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues

Abstract

Disclosed are some examples of systems, methods, apparatus and computer program products for scheduling jobs to process log files. In some implementations, a scheduler can be configured to identify a final state of a previously scheduled first job as a failure in execution by one or more processors. For instance, the first job may have been defined to process a first input log file saved to a database during a first timeframe of a sequence of timeframes of a designated interval. The scheduler can detect a second input log file saved to the database. The scheduler can then define a second job to process the first input log file and the second input log file and schedule the second job in association with a second timeframe of the sequence.

Claims

1. A system comprising:

a scheduler implemented using one or more processors of a server system, the scheduler configurable to cause;

identifying a final state of a first job previously scheduled by the scheduler as being a partial success or a partial failure in execution by one or more processors, the first job having been previously defined by the scheduler to process at least one first input log file saved to a database during a first one of a sequence of timeframes of a designated interval;

detecting at least one second input log file saved to the database and associated with the first timeframe;

defining a second job to include the at least one first input log file for processing and omit the at least one second input log file responsive to;

identifying the final state of the first job as being a partial success or a partial failure,

detecting the at least one second input log file, and

determining that a retry limit for processing the at least one first input log file has not been traversed; and

scheduling, in association with a second one of the sequence of timeframes, the second job for execution by one or more processors.

2. The system of claim 1, the scheduler further configurable to cause:

identifying a final state of the second job as being partially successful in execution by one or more processors;

detecting at least one third input log file saved to the database and associated with the first timeframe;

defining, responsive to identifying the final state of the second job as being partially successful and responsive to detecting the at least one third input log file, a third job to process the at least one first input log file and the at least one second input log file, the third job omitting the at least one third input log file; and

scheduling, in association with a third one of the sequence of timeframes, the third job for execution by one or more processors.

3. The system of claim 2, the scheduler further configurable to cause:

determining that a retry limit for processing a further input log file associated with the second job has not been traversed by a number of scheduled jobs.

4. The system of claim 2, the scheduler further configurable to cause:

identifying a final state of the third job as being successful in execution by one or more processors;

detecting at least one fourth input log file saved to the database and associated with the first timeframe;

defining, responsive to identifying the final state of the third job as being successful and responsive to detecting the at least one fourth input log file, a fourth job to process the at least one third input log file and the at least one fourth input log file; and

scheduling, in association with a fourth one of the sequence of timeframes, the fourth job for execution by one or more processors.

5. The system of claim 4, the scheduler further configurable to cause:

identifying an output log file resulting from the successful execution of the third job;

treating, responsive to identifying the output log file, the output log file as an immutable job object immediately accessible by computing devices of one or more designated tenants of a multi-tenant database system.

6. The system of claim 1, the scheduler further configurable to cause:

storing metadata in association with the second job, the metadata representing a plurality of parameters of the second job, at least one of the parameters indicating that the second job is a re-run of the first job.

7. The system of claim 1, wherein the scheduler is configurable to run at the designated interval plus an offset.

8. The system of claim 7, wherein the designated interval is one or more hours.

9. A computer program product comprising computer-readable program code capable of being executed by one or more processors when retrieved from a non-transitory computer-readable medium, the program code comprising instructions configurable to cause:

identifying a final state of a first job previously scheduled by a scheduler as being a partial success or a partial failure in execution by one or more processors, the first job having been previously defined by the scheduler to process at least one first input log file saved to a database during a first one of a sequence of timeframes of a designated interval;

detecting at least one second input log file saved to the database and associated with the first timeframe;

defining a second job to include the at least one first input log file for processing and omit the at least one second input log file responsive to;

identifyinq the final state of the first job as being a partial success or a partial failure,

detecting the at least one second input log file, and

determining that a retry limit for processing the at least one first input log file has not been traversed; and

scheduling, in association with a second one of the sequence of timeframes, the second job for execution by one or more processors.

10. The computer program product of claim 9, the instructions further configurable to cause:

identifying a final state of the second job as being partially successful in execution by one or more processors;

detecting at least one third input log file saved to the database and associated with the first timeframe;

defining, responsive to identifying the final state of the second job as being partially successful and responsive to detecting the at least one third input log file, a third job to process the at least one first input log file and the at least one second input log file, the third job omitting the at least one third input log file; and

scheduling, in association with a third one of the sequence of timeframes, the third job for execution by one or more processors.

11. The computer program product of claim 10, the instructions further configurable to cause:

determining that a retry limit for processing a further input log file associated with the second job has not been traversed by a number of scheduled jobs.

12. The computer program product of claim 10, the instructions further configurable to cause:

identifying a final state of the third job as being successful in execution by one or more processors;

detecting at least one fourth input log file saved to the database and associated with the first timeframe;

defining, responsive to identifying the final state of the third job as being successful and responsive to detecting the at least one fourth input log file, a fourth job to process the at least one third input log file and the at least one fourth input log file; and

scheduling, in association with a fourth one of the sequence of timeframes, the fourth job for execution by one or more processors.

13. The computer program product of claim 12, the instructions further configurable to cause:

identifying an output log file resulting from the successful execution of the third job;

treating, responsive to identifying the output log file, the output log file as an immutable job object immediately accessible by computing devices of one or more designated tenants of a multi-tenant database system.

14. The computer program product of claim 9, the instructions further configurable to cause:

storing metadata in association with the second job, the metadata representing a plurality of parameters of the second job, at least one of the parameters indicating that the second job is a re-run of the first job.

15. A method comprising:

identifying a final state of a first job previously scheduled by a scheduler as being a partial success or a partial failure in execution by one or more processors, the first job having been previously defined by the scheduler to process at least one first input log file saved to a database during a first one of a sequence of timeframes of a designated interval;

detecting at least one second input log file saved to the database and associated with the first timeframe;

defining a second job to include the at least one first input log file for processing and omit the at least one second input log file responsive to;

identifying the final state of the first job as being a partial success or a partial failure,

detecting the at least one second input log file, and

determining that a retry limit for processing the at least one first input log file has not been traversed; and

scheduling, in association with a second one of the sequence of timeframes, the second job for execution by one or more processors.

16. The method of claim 15, further comprising:

identifying a final state of the second job as being partially successful in execution by one or more processors;

detecting at least one third input log file saved to the database and associated with the first timeframe;

defining, responsive to identifying the final state of the second job as being partially successful and responsive to detecting the at least one third input log file, a third job to process the at least one first input log file and the at least one second input log file, the third job omitting the at least one third input log file; and

scheduling, in association with a third one of the sequence of timeframes, the third job for execution by one or more processors.

17. The method of claim 16, further comprising:

determining that a retry limit for processing a further input log file associated with the second job has not been traversed by a number of scheduled jobs.

18. The method of claim 16, further comprising:

identifying a final state of the third job as being successful in execution by one or more processors;

detecting at least one fourth input log file saved to the database and associated with the first timeframe;

defining, responsive to identifying the final state of the third job as being successful and responsive to detecting the at least one fourth input log file, a fourth job to process the at least one third input log file and the at least one fourth input log file; and

scheduling, in association with a fourth one of the sequence of timeframes, the fourth job for execution by one or more processors.

19. The method of claim 18, further comprising:

identifying an output log file resulting from the successful execution of the third job;

treating, responsive to identifying the output log file, the output log file as an immutable job object immediately accessible by computing devices of one or more designated tenants of a multi-tenant database system.

20. The method of claim 15, further comprising:

storing metadata in association with the second job, the metadata representing a plurality of parameters of the second job, at least one of the parameters indicating that the second job is a re-run of the first job.