Systems and methods for indexing and aggregating data records

• US 9,384,203 B1
• Filed: 06/09/2015
• Issued: 07/05/2016
• Est. Priority Date: 06/09/2015
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Seiver, Miles, Lemanowicz, Lucas
• Primary Examiner(s): Arjomandi, Noosha
• Application Number:
  US14/734,772
• Time in Patent Office: 392 Days
• Field of Search: 707/740
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54

Abstract

Computer-implemented systems and methods are disclosed for indexing and aggregating proxy server data. In accordance with some embodiments, the systems comprise a memory device configured to store a set of instructions, and one or more processors configured to execute the set of instructions to receive a plurality of log lines of proxy server data comprising a plurality of data components and index one or more of the plurality of data components for the plurality of log lines. The processors also generate a plurality of log data records, each including the one or more indexed data components, and generate a summarized data record by aggregating one or more log data records having at least one matching indexed data component.

Claims

1. A system for indexing and aggregating proxy server data, the system comprising:

a memory device configured to store a set of instructions; and

one or more processors configured to execute the set of instructions to;

receive a plurality of log lines of proxy server data comprising a plurality of data components;

index one or more of the plurality of data components for the plurality of log lines;

generate a set of structured log data records, each including the one or more indexed data components;

generate a set of first summarized data records, each first summarized data record of the set generated by aggregating one or more log data records of the set of log data records having at least one matching indexed data component, wherein each first summarized data record includes fewer indexed data components than a log data record; and

generate, after passage of a predetermined period of time since generation of the set of first summarized data records, a set of second summarized data records, each second summarized data record of the second set generated by aggregating one or more first summarized data records having a matching subset of the indexed data components, wherein each second summarized data record includes fewer indexed data components than a first summarized data record.

2. The system of claim 1, wherein the one or more processors are further configured to:

receive a search request relating to the at least one matching indexed data component; and

provide instructions to display information relating to the matching indexed data component based on a search of the first or second summarized data records.

3. The system of claim 1, wherein the one or more processors are further configured to include an additional data component in each first summarized data record based on the number of log data records aggregated into the first summarized data record.

4. The system of claim 1, wherein the one or more processors are further configured to generate the first set of summarized data records based on one or more log data records associated with a certain period of time.

5. The system of claim 1, wherein the at least one matching indexed data component includes a matching remote address or domain.

6. A computer-implemented method for indexing and aggregating a plurality of data records, the method comprising:

receiving a plurality of data records comprising a plurality of data components;

indexing one or more of the plurality of data components for the plurality of data records;

selecting one or more of the indexed data components for performing an aggregation on;

aggregating one or more of the plurality of data records having matching indexed data components of the selected indexed data components;

generating, for storage in place of the one or more aggregated data records, a first summarized data record including the matching indexed data components of the aggregated data records, the first summarized data record including fewer data components than a data record; and

generating, after passage of a predetermined period of time since generation of the first summarized data record, a second summarized data record by aggregating one or more first summarized data records having a matching subset of the selected indexed data components, the second summarized data record including fewer indexed data components than a first summarized data record;

wherein the method is performed using one or more processors.

7. The computer-implemented method of claim 6, further comprising:

receiving a search request relating to the at least one matching indexed data component; and

generating instructions to display information relating to the matching indexed data component based on a search of first or second summarized data records.

8. The computer-implemented method of claim 6, wherein the first summarized data record is generated to include an additional data component based on the aggregated data records.

9. The computer-implemented method of claim 8, wherein the additional data component corresponds to a total number of data records aggregated in the first summarized data record.

10. The computer-implemented method of claim 6, wherein the first summarized data record is generated based on one or more data records associated with a certain period of time.

11. A non-transitory computer-readable medium storing a set of instructions that are executable by one or more processors to cause the one or more processors to perform a method to index and aggregate a plurality of data records, the method comprising:

receiving a plurality of data records comprising a plurality of data components;

indexing one or more of the plurality of data components for the plurality of data records;

selecting one or more of the indexed data components for performing an aggregation on;

aggregating one or more of the plurality of data records having matching indexed data components of the selected indexed data components;

generating, for storage in place of the one or more aggregated data records, a first summarized data record including the matching indexed data components of the aggregated data records, the first summarized data record including fewer data components than a data record; and

generating, after passage of a predetermined period of time since generation of the first summarized data record, a second summarized data record by aggregating one or more first summarized data records having a matching subset of the selected indexed data components, the second summarized data record including fewer indexed data components than a first summarized data record.

12. The non-transitory computer-readable medium of claim 11, wherein the method further comprises:

receiving a search request relating to the at least one matching indexed data component; and

providing instructions to display information relating to the matching indexed data component based on a search of first or second summarized data records.

13. The non-transitory computer-readable medium of claim 11, wherein the first summarized data record is generated to include an additional data component based on the number of aggregated data records.

14. The non-transitory computer-readable medium of claim 11, wherein the first summarized data record is generated based on one or more data records associated with a period of time.

15. The non-transitory computer-readable medium of claim 11, wherein the at least one matching indexed data component includes a matching remote address or domain.