Data pipeline monitoring

• US 9,678,850 B1
• Filed: 08/01/2016
• Issued: 06/13/2017
• Est. Priority Date: 06/10/2016
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Rickard, Jesse, Maag, Peter, Newman, Jared, Mecocci, Giulio, Subbanarasimhia, Harish, Dumitran, Adrian Marius, Skrodzki, Andrzej, Scheinerman, Jonah, Slonim, Gregory, Antihi, Alexandru Viorel
• Primary Examiner(s): Roni, Syed
• Application Number:
  US15/225,738
• Time in Patent Office: 316 Days
• Field of Search: None
• CPC Class Codes: G06F 11/3024 where the computing system component is a central processing unit [CPU]
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/3093 Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
• CPC Class Codes: G06F 11/328 Computer systems status display G06F11/327 takes precedence
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs

Abstract

A method and system for data pipeline monitoring receives an event data object and a current status data object from one or more subsystems of a pipeline. The system analyzes the event data object and the current status data object to determine a first and second validation value. The system, in response to determining that either the first or second validation value is not valid, sends a notification.

Claims

1. A method comprising:

receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;

receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;

analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;

analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation criteria to the current status data object;

in response to determining that the first validation value is not valid, sending a first notification regarding the event data object;

in response to determining that the second validation value is not valid, sending a second notification regarding the current status data object; and

wherein the method is performed using one or more processors.

2. The method of claim 1, wherein the first validation criteria comprises criteria specified in a script.

3. The method of claim 1, wherein a source of the event data object is an event log.

4. The method of claim 1, wherein the current status data object comprises information regarding one or more of:

disk space, memory usage, and central processing unit (CPU) usage.

5. The method of claim 1, wherein:

receiving an event data object that indicates a historical status of one or more subsystems of a pipeline is scheduled based on a first preconfigured schedule; and

receiving a current status data object that indicates a current status of one or more subsystems of a pipeline is scheduled based on the first preconfigured schedule.

6. The method of claim 1, further comprising:

customizing a format of the first notification based on the first validation value; and

customizing a format of the second notification based on the second validation value.

7. The method of claim 1, further comprising:

customizing a first recipient of the first notification based on the first validation value; and

customizing a second recipient of the second notification based on the second validation value.

8. The method of claim 1, further comprising displaying a user interface based on the event data object, the current status data object, the first validation value, the second validation value, the first notification and the second notification.

9. The method of claim 8, wherein the user interface comprises a summary card summarizing the first validation value and the second validation value for a particular subsystem of the one or more subsystems.

10. A system, comprising:

a plurality of computers;

one or more non-transitory data storage media in the plurality of computers storing logic comprising one or more sequences of instructions which when executed cause the computers to perform;

receiving an event data object that indicates a historical status of one or more subsystems of a pipeline;

receiving a current status data object that indicates a current status of one or more subsystems of a pipeline;

analyzing the event data object to determine a first validation value of the event data object by applying a first validation criteria to the event data object;

analyzing the current status data object to determine a second validation value of the current status data object by applying a second validation value to the current status data object;

in response to determining that the first validation value is not valid, sending a first notification regarding the event data object; and

in response to determining that the second validation value is not valid, sending a second notification regarding the current status data object.

11. The system of claim 10, wherein the first validation criteria comprises criteria specified in a script.

12. The system of claim 10, wherein a source of the event data object is an event log.

13. The system of claim 10, wherein the current status data object comprises information regarding one or more of:

disk space, memory usage, and central processing unit (CPU) usage.

14. The system of claim 10, wherein:

the instructions for receiving an event data object that indicates a historical status of one or more subsystems of a pipeline is scheduled based on a first preconfigured schedule; and

the instructions receiving a current status data object that indicates a current status of one or more subsystems of a pipeline is scheduled based on the first preconfigured schedule.

15. The system of claim 10, further comprising instructions for:

customizing a format of the first notification based on the first validation value; and

customizing a format of the second notification based on the second validation value.

16. The system of claim 10, further comprising instructions for:

customizing a first recipient of the first notification based on the first validation value; and

customizing a second recipient of the second notification based on the second validation value.

17. The system of claim 10, further comprising instructions for displaying a user interface based on the event data object, the current status data object, the first validation value, the second validation value, the first notification and the second notification.

18. The system of claim 17, wherein the user interface comprises a summary card summarizing the first validation value and the second validation value for a particular subsystem of the one or more subsystems.