Systems and methods for indexing and aggregating data records

• US 9,922,113 B2
• Filed: 07/01/2016
• Issued: 03/20/2018
• Est. Priority Date: 06/09/2015
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Seiver, Miles, Lemanowicz, Lucas
• Primary Examiner(s): Arjomandi, Noosha
• Application Number:
  US15/201,282
• Publication Number:
  US 20160364470A1
• Time in Patent Office: 627 Days
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54

Abstract

Computer-implemented systems and methods are disclosed for indexing and aggregating proxy server data. In accordance with some embodiments, the systems comprise a memory device configured to store a set of instructions, and one or more processors configured to execute the set of instructions to receive a plurality of log lines of proxy server data comprising a plurality of data components and index one or more of the plurality of data components for the plurality of log lines. The processors also generate a plurality of log data records, each including the one or more indexed data components, and generate a summarized data record by aggregating one or more log data records having at least one matching indexed data component.

Claims

1. A system for reducing storage space requirements by aggregating similar structured data records, the system comprising:

a memory device configured to store a set of instructions; and

one or more processors configured to execute the set of instructions to;

access a plurality of structured data records each comprising a plurality of data values in a corresponding plurality of data fields, the data fields including at least a first data field and a second data field;

identify those of the plurality of structured data records having a first data value for each of the corresponding first data fields, wherein second data values of corresponding second data fields of the identified plurality of structured data records are unique;

determine a quantity of the identified structured data records having the first data value for each of the corresponding first data fields; and

generate, for storage in place of the identified structured data records, a summarized data record indicating the corresponding first data value common among each of the identified structured data records and the quantity of identified structured data records, wherein the summarized data record does not include the second data values of the identified structured data records.

2. The system of claim 1, wherein, the quantity of identified structured data records is included in an additional data field of the summarized data record.

3. The system of claim 1, wherein the summarized data record further comprises an aggregate value of each of a particular data field of the identified structured data records.

4. The system of claim 1, wherein the one or more processors are further configured to:

receive a search request relating to at least one of data fields; and

provide instructions to display information relating to the request based on a search of summarized data records.

5. The system of claim 1, wherein the plurality of structured data records summarized in the summarized data record are limited to those associated with a certain period of time.

6. The system of claim 1, wherein the one or more processors are further configured to:

identify those of a plurality of summarized data records each a common value for a first data field; and

generate, for storage in place of the identified summarized data records, a second summarized data record indicating the common value.

7. The system of claim 1, wherein the plurality of structured data records correspond to a plurality of indexed log lines of proxy server data, and the first data field corresponds to a remote address or domain.

8. A computer-implemented method for comprising:

access a plurality of structured data records, each comprising a plurality of data values in a corresponding plurality of data fields, the data fields including at least a first data field and a second data field;

identify those of the plurality of structured data records having a first data value for each of the corresponding first data fields, wherein second data values of corresponding second data fields of the identified plurality of structured data records are unique;

determine a quantity of the identified structured data records having the first data value for each of the corresponding first data fields; and

generate, for storage in place of the identified structured data records, a summarized data record, indicating the corresponding first data value common among each of the identified structured data records and the quantity of identified structured data record, wherein the summarized data record does not include the second data values of the identified structured data records;

wherein the method is performed using one or more processors.

9. The computer-implemented method of claim 8, wherein the quantity of identified structured data records is included in an additional data field of the summarized data record.

10. The computer-implemented method of claim 8, wherein the summarized data record further comprises an aggregate value of each of a particular data field of the identified structured data records.

11. The computer-implemented method of claim 8, further comprising:

receiving a search request relating to at least one of the data fields; and

providing instructions to display information relating to the request based on a search of summarized data records.

12. The computer-implemented method of claim 8, wherein the plurality of structured data records summarized in the summarized data record are limited to those associated with a certain period of time.

13. The computer-implemented method of claim 8, further comprising:

identifying those of a plurality of summarized data records each a common value for a first data fields; and

generating, for storage in place of the identified summarized data records, a second summarized data record indicating the common value.

14. The computer-implemented method of claim 8, wherein the plurality of structured data records correspond to a plurality of indexed log lines of proxy server data, and the first data field corresponds to a remote address or domain.

15. A non-transitory computer-readable medium storing a set of instructions that are executable by one or more processors to cause a computing system to:

access a plurality of structured data records, each comprising a plurality of data values in a corresponding plurality of data fields, the data fields including at least a first data field and a second data field;

identify those of the plurality of structured data records having a first data value for each of the corresponding first data fields, wherein second data values of corresponding second data fields of the identified plurality of structured data records are unique;

determine a quantity of the identified structured data records having the first data value for each of the corresponding first data fields; and

generate, for storage in place of the identified structured data records, a summarized data record indicating the corresponding first data value common among each of the identified structured data records and the quantity of identified structured data record, wherein the summarized data record does not include the second data values of the identified structured data records.

16. The non-transitory computer-readable medium of claim 15, wherein the summarized data record is generated to include an additional data field indicative of the quantity of identified structured data records.

17. The non-transitory computer-readable medium of claim 15, wherein the method further comprises:

identifying those of a plurality of summarized data records each a common value for a first data field; and

generating, for storage in place of the identified summarized data records, a second summarized data record indicating the common value.