Multiple sensor session and log information compression and correlation system

• US 10,698,927 B1
• Filed: 08/08/2017
• Issued: 06/30/2020
• Est. Priority Date: 08/30/2016
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Chin, Daniel, Dries, Daniel
• Primary Examiner(s): Ruiz, Angelica
• Application Number:
  US15/672,069
• Time in Patent Office: 1,057 Days
• Field of Search: 707600-831, 707899
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/2322 using timestamps
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 7/08 Sorting, i.e. grouping record carriers in numerical or other ordered sequence according to the classification of at least some of the information they carry by merging two or more sets of carriers in ordered sequence G06F7/16
• CPC Class Codes: G06F 9/547 Remote procedure calls [RPC]; Web services

Abstract

Methods, systems and apparatus, including computer programs encoded on computer storage media for compressing sensor log information. One of the methods includes accessing log information maintained in one or more databases, the log information being generated in response to actions associated with entities, and the log information indicative of respective sessions for which one or more logs were generated, each log indicating an entity. Log information is grouped according to entity. One or more logs associated with respective sessions based on the grouped log information. Compressed logs are generated from logs associated with respective sessions based on compression rules.

Claims

1. A method comprising:

by a system comprising one or more processors,

accessing log information maintained in one or more databases, the log information being generated in response to actions associated with entities, the log information including, at least, information indicative of respective sessions for which one or more logs were generated, the actions being indicative of events associated with sessions, each log including a plurality of columns and indicating an entity in a first column of the plurality of columns;

grouping log information according to entity;

determining, for respective entities based on the grouped log information, one or more logs associated with respective sessions; and

generating, from logs associated with each session, a compressed log based on one or more compression rules, wherein generating the compressed log associated with the each session comprises;

identifying a plurality of actions associated with the each session, the plurality of actions being included in a plurality of rows of a second column of the logs associated with the each session, and

concatenating the plurality of actions into a single row of the plurality of rows of the second column, wherein remaining rows of the plurality of rows of the second column are discarded.

2. The method of claim 1, wherein the log information is included in one or more database tables, and wherein the method further comprises:

updating the one or more database tables to include the compressed logs, thereby reducing a size associated with the database tables.

3. The method of claim 1, wherein each log indicates a timestamp associated with its generation, and wherein grouping log information according to entity comprises:

grouping logs according to entity; and

for each entity, sorting log information associated with the entity according to timestamp.

4. The method of claim 1, wherein a compression rule specifies a rule associated with combining, discarding, or selecting, information indicated in the logs associated a session.

5. The method of claim 1, wherein the log information is included in one or more database tables, and wherein at least two logs associated with a particular session are separated in the database tables by hundreds or thousands of logs.

6. The method of claim 1, further comprising:

receiving, via application programing interface (API) calls, requests for compressed logs, the requests being received over one or more networks by an outside system associated with the entities; and

providing, to the outside system, the compressed logs, such that the outside system can perform corrective actions in response to analyzing the compressed logs.

7. The method of claim 1, further comprising:

accessing information indicating subscriptions to types of information by user devices;

identifying, based on the compressed logs, a particular compressed log that includes a type of information to which a particular user device subscribes; and

generating notification information for providing to the user device, the notification information automatically activating the particular user device to perform one or more corrective actions.

8. A system comprising one or more computers and computer storage media storing instructions that when executed by the one or more computers, cause the one or more computers to perform operations comprising:

accessing log information maintained in one or more databases, the log information being generated in response to actions associated with entities, the log information including, at least, information indicative of respective sessions for which one or more logs were generated the actions being indicative of events associated with sessions, each log including a plurality of columns and indicating an entity in a first column of the plurality of columns;

grouping log information according to entity;

determining, for respective entities based on the grouped log information, one or more logs associated with respective sessions; and

generating, from logs associated with each session, a compressed log based on one or more compression rules, wherein generating the compressed log associated with the each session comprises;

identifying a plurality of actions associated with the each session, the plurality of actions being included in a plurality of rows of a second column of the logs associated with the each session, and

concatenating the plurality of actions into a single row of the plurality of rows of the second column, wherein remaining rows of the plurality of rows of the second column are discarded.

9. The system of claim 8, wherein the log information is included in one or more database tables, and wherein the operations further comprise:

updating the one or more database tables to include the compressed logs, thereby reducing a size associated with the database tables.

10. The system of claim 8, wherein each log indicates a timestamp associated with its generation, and wherein grouping log information according to entity comprises:

grouping logs according to entity; and

for each entity, sorting log information associated with the entity according to timestamp.

11. The system of claim 8, wherein a compression rule specifies a rule associated with combining, discarding, or selecting, information indicated in the logs associated a session.

12. The system of claim 8, wherein the log information is included in one or more database tables, and wherein at least two logs associated with a particular session are separated in the database tables by hundreds or thousands of logs.

13. The system of claim 8, wherein the operations further comprise:

receiving, via application programing interface (API) calls, requests for compressed logs, the requests being received over one or more networks by an outside system associated with the entities; and

providing, to the outside system, the compressed logs, such that the outside system can perform corrective actions in response to analyzing the compressed logs.

14. The system of claim 8, wherein the operations further comprise:

accessing information indicating subscriptions to types of information by user devices;

identifying, based on the compressed logs, a particular compressed log that includes a type of information to which a particular user device subscribes; and

generating notification information for providing to the user device, the notification information automatically activating the particular user device to perform one or more corrective actions.

15. Non-transitory computer storage media storing instructions that when executed by a system of one or more computers cause the system to perform operations comprising:

accessing log information maintained in one or more databases, the log information being generated in response to actions associated with entities, the log information including, at least, information indicative of respective sessions for which one or more logs were generated, the actions being indicative of events associated with sessions, each log including a plurality of columns and indicating an entity in a first column of the plurality of columns;

grouping log information according to entity;

determining, for respective entities based on the grouped log information, one or more logs associated with respective sessions; and

generating, from logs associated with each session, a compressed log based on one or more compression rules, wherein generating the compressed log associated with the each session comprises;

identifying a plurality of actions associated with the each session, the plurality of actions being included in a plurality of rows of a second column of the logs associated with the each session, and

concatenating the plurality of actions into a single row of the plurality of rows of the second column, wherein remaining rows of the plurality of rows of the second column are discarded.

16. The computer-storage media of claim 15, wherein the log information is included in one or more database tables, and wherein the operations further comprise:

updating the one or more database tables to include the compressed logs, thereby reducing a size associated with the database tables.

17. The computer-storage media of claim 15, wherein each log indicates a timestamp associated with its generation, and wherein grouping log information according to entity comprises:

grouping logs according to entity; and

for each entity, sorting log information associated with the entity according to timestamp.

18. The computer-storage media of claim 15, wherein a compression rule specifies a rule associated with combining, discarding, or selecting, information indicated in the logs associated a session.

19. The computer-storage media of claim 15, wherein the operations further comprise:

receiving, via application programing interface (API) calls, requests for compressed logs, the requests being received over one or more networks by an outside system associated with the entities; and

providing, to the outside system, the compressed logs, such that the outside system can perform corrective actions in response to analyzing the compressed logs.

20. The computer-storage media of claim 15, wherein the operations further comprise:

accessing information indicating subscriptions to types of information by user devices;

identifying, based on the compressed logs, a particular compressed log that includes a type of information to which a particular user device subscribes; and

generating notification information for providing to the user device, the notification information automatically activating the particular user device to perform one or more corrective actions.