Data processing platform monitoring

• US 10,365,964 B1
• Filed: 05/31/2018
• Issued: 07/30/2019
• Est. Priority Date: 05/31/2018
• Status: Active Grant

• Current Assignee: Capital One Financial Corporation, Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Subburasu, Ragupathi, Gupta, Mayur, Palamari, Ravi Kiran
• Primary Examiner(s): Bonzo, Bryce P
• Assistant Examiner(s): Arcos, Jeison C
• Application Number:
  US15/994,640
• Time in Patent Office: 425 Days
• CPC Class Codes: G06F 11/0712 in a virtual computing platform, e.g. logically partitioned systems
• CPC Class Codes: G06F 11/0745 in an input/output transactions management context input/output processing in general G06F13/00
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 2209/5013 Request control
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals

Abstract

A device may receive error data indicating that an error occurred, the error being associated with a data processing job scheduled to be performed by a data processing platform. The device may identify input data for the data processing job associated with the error and determine that the error is based on the data processing platform not receiving the input data. In addition, the device may determine a location of the input data and determine a measure of priority associated with the data processing job. Based on the location of the input data and the measure of priority, the device may perform an action to correct the error.

Claims

1. A method, comprising:

receiving, by a device, error data indicating that an error occurred, the error being associated with a data processing job scheduled to be performed by a data processing platform;

identifying, by the device, input data for the data processing job associated with the error;

identifying, by the device, an expected format for the input data;

determining, by the device and based on a regular expression, that the error is further based on a format of the input data not matching the expected format;

determining, by the device and based on the error, a location of the input data;

determining, by the device, a measure of priority associated with the data processing job; and

performing, by the device, an action to correct the error, based on the location of the input data and the measure of priority.

2. The method of claim 1, wherein:

determining the location of the input data includes;

determining that the input data is stored in a data storage device accessible to the device; and

performing the action based on the location of the input data and the measure of priority includes;

providing the data storage device with a request to provide the input data to the data processing platform.

3. The method of claim 1, wherein:

determining the location of the input data includes;

determining that the input data is stored by a third party device and not accessible to the device; and

performing the action based on the location of the input data and the measure of priority includes;

providing the third party device with a request to provide the input data to the data processing platform.

4. The method of claim 1, wherein:

determining the location of the input data includes;

determining that the input data is stored in a data storage device accessible to the data processing platform; and

performing the action based on the location of the input data and the measure of priority includes;

providing the data processing platform with data specifying the location of the input data.

5. The method of claim 1, wherein performing the action based on the location of the input data and the measure of priority includes:

providing, based on the measure of priority meeting a threshold measure of priority, another device with data identifying the location of the input data to permit the other device to escalate obtaining the input data based on the data identifying the location of the input data.

6. The method of claim 1, wherein determining the location of the input data includes:

determining the location of the input data by searching for data that matches the expected format.

7. The method of claim 1, further comprising:

monitoring an error log associated with the data processing platform, wherein the error data is received from the error log.

8. A device comprising:

one or more memories; and

one or more processors, communicatively coupled to the one or more memories, to;

receive error data indicating that a data processing platform encountered an error, the error being associated with a data processing job;

identify an intended format for input data associated with the data processing job;

determine, using a regular expression, that the error is based on a fault with a format of the input data, the fault being determined based on the format of the input data not matching the intended format;

identify a faulty portion of the input data based on the intended format, the faulty portion including the fault;

generate corrected input data based on the intended format and the faulty portion; and

provide the data processing platform with data that causes the data processing platform to perform the data processing job using the corrected input data.

9. The device of claim 8, wherein the one or more processors, when receiving the error data, are to:

monitor a data structure that stores error log data produced by the data processing platform to identify the error data.

10. The device of claim 9, wherein the one or more processors, when monitoring the data structure that stores the error log data produced by the data processing platform, are to:

periodically read the data structure.

11. The device of claim 8, wherein the error data is received from a job scheduler device associated with the data processing platform.

12. The device of claim 8, wherein the one or more processors, when generating corrected input based on the intended format and the faulty portion, are to:

generate, based on the input data, a corrected input file associated with the intended format.

13. The device of claim 8, wherein the one or more processors, when generating corrected input based on the intended format and the faulty portion, are to:

remove, from the input data, the faulty portion.

14. A non-transitory computer-readable medium storing instructions, the instructions comprising:

one or more instructions that, when executed by one or more processors, cause the one or more processors to;

monitor a data processing platform;

determine that the data processing platform is scheduled to perform a data processing job;

determine, based on a regular expression, that input data needed for the data processing job is missing or incorrect, the determination that the input data needed for the data processing job is missing or incorrect being based on a format of the input data failing to match an intended format;

determine a location of the input data;

identify, based on the location of the input data, corrected input data that corrects the input data that is missing or incorrect; and

cause the corrected input data to be provided to the data processing platform to permit the data processing platform to perform the data processing job.

15. The non-transitory computer-readable medium of claim 14, wherein the one or more instructions that, when executed by one or more processors, cause the one or more processors to monitor the data processing platform, cause the one or more processors to:

determine an expected input data location associated with the data processing job; and

prior to a scheduled execution time for the data processing job, monitor the expected input data location for the input data.

16. The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to:

search for the input data at an expected input data location, the expected input data location being previously configured for the data processing job; and

determine that the data processing platform encountered an error based on the input data not being available at the expected input data location.

17. The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to:

provide the data processing platform with data identifying the location of the input data; and

provide the data processing platform with data that causes the data processing platform to perform the data processing job based on the location of the input data.

18. The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to:

provide, to a device associated with the location of the input data, a request that causes the device to make the input data available at an expected input data location;

identify a measure of priority associated with the data processing job; and

provide, based on the measure of priority, the data processing platform with data that causes the data processing platform to perform the data processing job.

19. The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to:

identify a measure of priority associated with the data processing job; and

provide, based on the measure of priority, another device with data identifying the location of the input data to permit the other device to escalate obtaining the input data based on the data identifying the location of the input data.

20. The non-transitory computer-readable medium of claim 14, wherein the one or more instructions, when executed by one or more processors, further cause the one or more processors to:

identify a measure of priority associated with the data processing job;

determine a delay based on the measure of priority; and

schedule, based on the delay, the data processing job.