Linking related events for various devices and services in computer log files on a centralized server

• US 10,698,756 B1
• Filed: 09/12/2018
• Issued: 06/30/2020
• Est. Priority Date: 12/15/2017
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Abdelsalam, Mahmoud, Santos, Eric, DeArment, Greg, Wu, Grant, Ross, James, Simons, Kevin, Xu, Guodong, Wang, Peter, Niemi, Richard, Lakshmanan, Vivek
• Primary Examiner(s): Chu, Gabriel
• Application Number:
  US16/129,626
• Time in Patent Office: 657 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/0781 Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3068 where the reporting involves data format conversion
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/366 using diagnostics G06F11/0703 takes precedence
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 21/6245 Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 21/6254 by anonymising data, e.g. decorrelating personal data from the owner's identification
• CPC Class Codes: G06F 2201/835 Timestamp

Abstract

A system with an interactive user interface for users to view and interact with sanitized log data received from a plurality of hosts, such as those associated with various services of an organization. The system may receive from hosts log files and/or metadata that have been filtered by agents executing on the respective hosts to remove or anonymize any sensitive or confidential information prior to transmission to the system. In some embodiments the system does further filtering of the sanitized data. Received sanitized data is parsed, indexed, and/or otherwise processed for optimal searching, and stored in a log pipeline. The system causes display of an electronic visualization interface comprising a dynamic electronic search configured to receive an indication of various log search criteria, such as an error or trace identifier, that are used to identify matching log files meeting the provided criteria, such as log files associated with services executed on different hosts.

Claims

1. A computing system for dynamically generating a dynamic electronic search system, the system comprising:

one or more computer readable storage devices configured to store a plurality of computer executable instructions; and

one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the plurality of computer executable instructions in order to cause the computing system to;

electronically receive, via one or more networks, sanitized data from a plurality of remote hosts, wherein the sanitized data comprises a plurality of log files each including unique identifiers of associated software services, wherein the remote hosts are configured to apply rules to determine data that is uploaded to a log pipeline, wherein at least some unique identifiers of software services are included in multiple log files associated with related events performed by each of multiple software services;

store, in a log pipeline comprising one or more databases, the plurality of log files;

parsing the log files into log attributes, the log attributes including at least time stamps and unique identifiers of associated software services of the respective log files;

indexing the log files, including the log attributes, into an indexed searching platform; and

cause display of an electronic visualization interface, the electronic visualization interface comprising a dynamic electronic search configured to receive an indication of a unique identifier and identifying, based on the indexed searching platform, two or more matching log files having the same unique identifier from different hosts.

2. The computing system of claim 1, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

remove from the parsed log files information deemed to be sensitive.

3. The computing system of claim 1, wherein the sanitized data includes log data and metadata from one or more of the remote hosts.

4. The computing system of claim 3, wherein the metadata comprises at least a hostname of the host.

5. The computing system of claim 3, wherein the metadata comprises at least an IP address.

6. The computing system of claim 1, wherein each of the remote hosts are configured to format the sanitized data into a normalized format.

7. The computing system of claim 1, wherein the indexed attributes comprise one or more of log freshness, error identifiers, trace identifiers, service identifiers, originating service, originating host, originating log, service version, error priority level, and magnitude of the error.

8. The computing system of claim 1, wherein the electronic visualization interface further comprises a customizable viewing pane configured to enable modifications to the display of the sanitized data.

9. The computing system of claim 1, wherein the electronic visualization interface further comprises a customizable viewing pane configured to enable customizations to the viewing pane comprising adding columns, removing columns, adding viewable data, and removing viewable data.

10. The computing system of claim 1, wherein the same unique identifier is a particular error instance ID.

11. A method performed by at least one electronic device comprising one or more processors, the method comprising:

electronically receive, via one or more networks, sanitized data from a plurality of remote hosts, wherein the sanitized data comprises a plurality of log files each including unique identifiers of associated software services, wherein the remote hosts are configured to apply rules to determine data that is uploaded to a log pipeline, wherein at least some unique identifiers of software services are included in multiple log files associated with related events performed by each of multiple software services;

store, in a log pipeline comprising one or more databases, the plurality of log files;

parsing the log files into log attributes, the log attributes including at least time stamps and unique identifiers of associated software services of the respective log files;

indexing the log files, including the log attributes, into an indexed searching platform; and

cause display of an electronic visualization interface, the electronic visualization interface comprising a dynamic electronic search configured to receive an indication of a unique identifier and identifying, based on the indexed searching platform, two or more matching log files having the same unique identifier from different hosts.

12. The method of claim 11, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

remove from the parsed log files information deemed to be sensitive.

13. The method of claim 11, wherein the sanitized data includes log data and metadata from one or more of the remote hosts.

14. The method of claim 11, wherein the electronic visualization interface further comprises a customizable viewing pane configured to enable modifications to the display of the sanitized data.

15. The method of claim 11, wherein the electronic visualization interface further comprises a customizable viewing pane configured to enable customizations to the viewing pane comprising adding columns, removing columns, adding viewable data, and removing viewable data.

16. The method of claim 11, wherein the unique identifier and same unique identifier is a particular error instance ID.

17. A non-transitory computer-readable medium storing a set of instructions that are executable by one or more electronic devices, each having one or more processors, to cause the one or more electronic devices to perform a method, the method comprising:

electronically receive, via one or more networks, sanitized data from a plurality of remote hosts, wherein the sanitized data comprises a plurality of log files each including unique identifiers of associated software services, wherein the remote hosts are configured to apply rules to determine data that is uploaded to a log pipeline, wherein at least some unique identifiers of software services are included in multiple log files associated with related events performed by each of multiple software services;

store, in a log pipeline comprising one or more databases, the plurality of log files;

parsing the log files into log attributes, the log attributes including at least time stamps and unique identifiers of associated software services of the respective log files;

indexing the log files, including the log attributes, into an indexed searching platform; and

cause display of an electronic visualization interface, the electronic visualization interface comprising a dynamic electronic search configured to receive an indication of a unique identifier and identifying, based on the indexed searching platform, two or more matching log files having the same unique identifier from different hosts.

18. The computer-readable medium of claim 17, wherein the one or more hardware computer processors are further configured to execute the plurality of computer executable instructions to cause the computer system to:

remove from the parsed log files information deemed to be sensitive.

19. The computer-readable medium of claim 17, wherein the sanitized data includes log data and metadata from one or more of the remote hosts.

20. The computer-readable medium claim 17, wherein the electronic visualization interface further comprises a customizable viewing pane configured to enable modifications to the display of the sanitized data.