SYSTEM AND METHOD FOR INVESTIGATING LARGE AMOUNTS OF DATA

• US 20120330908A1
• Filed: 06/23/2011
• Published: 12/27/2012
• Est. Priority Date: 06/23/2011
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Fischer, Chris, Stowe, Geoffrey, George, Paul, Bingham, Eli, Hill, Rosco
• Granted Patent: US 8,799,240 B2
• Time in Patent Office: Days
• US Class Current: 707/693
• CPC Class Codes: G06F 11/2025 using centralised failover control functionality
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/17 Details of further file system functions
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/902 using more than one table in sequence, i.e. systems with three or more layers
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 17/00 Digital computing or data processing equipment or methods, specially adapted for specific functions information retrieval, database structures or file system structures therefor G06F16/00

Abstract

A data analysis system is proposed for providing fine-grained low latency access to high volume input data from possibly multiple heterogeneous input data sources. The input data is parsed, optionally transformed, indexed, and stored in a horizontally-scalable key-value data repository where it may be accessed using low latency searches. The input data may be compressed into blocks before being stored to minimize storage requirements. The results of searches present input data in its original form. The input data may include access logs, call data records (CDRs), e-mail messages, etc. The system allows a data analyst to efficiently identify information of interest in a very large dynamic data set up to multiple petabytes in size. Once information of interest has been identified, that subset of the large data set can be imported into a dedicated or specialized data analysis system for an additional in-depth investigation and contextual analysis.

Claims

1. A method comprising:

obtaining a search criterion;

using the search criterion to obtain one or more first values from a first-key value family of a key-value data repository;

using the one or more first values to obtain one or more compressed values from a second key-value family of the key-value data repository;

uncompressing the one or more compressed values to produce one or more uncompressed values;

using the one or more first values to identify one or more portions of the one or more uncompressed values;

returning the one or more portions of the one or more uncompressed values as search results;

wherein the method is performed by one or more computing devices.

2. The method of claim 1:

wherein the first key-value family comprises a first set of unique keys, each key in the first set of unique keys mapping to one or more values;

wherein the second key-value family comprises a second set of unique keys, each key in the second set of unique keys mapping to at least one compressed value.

3. The method of claim 1, wherein at least one of the one or more first values comprises an identifier of a compressed value of the one or more compressed values.

4. The method of claim 1, wherein at least one of the one or more first values comprises a key of the second key-value family.

5. The method of claim 1, wherein at least one of the one or more first values comprises information identifying a byte sequential portion of one of the one or more uncompressed values.

6. The method of claim 1, wherein at least one of one or more portions is a byte sequential portion of an uncompressed value of the one or more uncompressed values.

7. The method of claim 1, further comprising:

receiving a search parameter from a user;

deriving the search criterion from the search parameter.

8. The method of claim 7, wherein deriving includes at least one of transforming, converting, encoding, and canonicalizing the search parameter to produce the search criterion.

9. The method of claim 1, wherein each key of the first key-value family is unique at least amongst all keys of the first key-value family.

10. The method of claim 1, wherein the first key-value family comprises at least one million unique keys.

11. The method of claim 1, wherein using the search criterion to obtain the one or more first values includes selecting a key from the first key-value family that equals the search criterion.

12. The method of claim 1:

wherein the key-value data repository comprises a cluster of a plurality of computing nodes;

wherein at least one key of the first key-value family is mastered by at least one node of the plurality nodes and at least one other key of the first key-value family is mastered by at least one other node of the plurality of nodes;

wherein each and every node of the cluster of nodes is configured to obtain values for any key of the first key-value family.

20. The method of claim 1, wherein at least one of the one or more second values comprises an identifier of a compressed value of the one or more compressed values.

21. The method of claim 1, further comprising:

receiving a search parameter from a user;

deriving the first search criterion from the search parameter.

22. The method of claim 21, wherein deriving includes at least one of transforming, converting, encoding, and canonicalizing the search parameter to produce the first search criterion.

23. The method of claim 1, wherein each key of the first key-value family is unique at least amongst all keys of the first key-value family.

24. The method of claim 1, wherein the first key-value family comprises at least one million unique keys.

39. One or more non-transitory computer-readable media storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method of claim 1.

13. A method comprising:

obtaining a first search criterion and one or more second search criteria;

using the first search criterion and the second search criteria to obtain one or more first values from a first-key value family of a key-value data repository;

using the one or more first values to obtain one or more second values from a second key-value family of the key-value data repository;

using the one or more second values to obtain one or more compressed values from a third key-value family of the key-value data repository;

returning one or more uncompressed portions of the one or more compressed values as search results;

wherein the method is performed by one or more computing devices.

14. The method of claim 13, further comprising:

uncompressing the one or more compressed values to produce one or more uncompressed values;

using the one or more second values to identify the one or more uncompressed portions from the one or more uncompressed values.

15. The method of claim 14, wherein at least one of the one or more second values comprises information identifying a byte sequential portion of one of the one or more uncompressed values.

16. The method of claim 14, wherein at least one of the one or more portions is a byte sequential portion of an uncompressed value of the one or more uncompressed values.

17. The method of claim 13, further comprising:

generating at least one of the second criteria using the first criterion and a search parameter obtained from a user.

18. The method of claim 13, wherein using the first search criterion and the second search criteria to obtain the one or more first values comprises selecting all values of a key of the first key-value family that equals the first search criterion where the selected values satisfy the second search criteria.

19. The method of claim 13:

wherein the first key-value family comprises a first set of unique keys and a set of values to which the first set of unique keys are mapped;

wherein the second key-value family comprises a second set of unique keys and a set of values to which the second set of unique keys are mapped;

wherein the third key-value family comprises a third set of unique keys and a set of values to which the third set of unique keys are mapped;

wherein at least one value of each key in the first set of unique keys matches a key of the second set of unique keys;

wherein at least one value of each key in the second set of unique keys matches a key of the third set of unique keys.

25. The method of claim 13:

wherein the key-value data repository comprises a cluster of a plurality of computing nodes;

wherein at least one key of the first key-value family is mastered by at least one node of the plurality nodes and at least one other key of the first key-value family is mastered by at least one other node of the plurality of nodes;

wherein each and every node of the cluster of nodes is configured to obtain values for any key of the first key-value family.

40. One or more non-transitory computer-readable media storing one or more sequences of instructions which, when executed by one or more processors, causes the one or more processors to perform the method of claim 13.

26. A system comprising:

a key-value data repository comprising a first key-value family and a second key-value family;

a search mechanism comprising one or more processors and configured to;

obtain a search criterion;

use the search criterion to obtain one or more first values from the first-key value family;

use the one or more first values to obtain one or more compressed values from the second key-value family;

uncompress the one or more compressed values to produce one or more uncompressed values;

use the one or more first values to identify one or more portions of the one or more uncompressed values;

return the one or more portions of the one or more uncompressed values as search results.

27. The system of claim 26:

wherein the first key-value family comprises a first set of unique keys, each key in the first set of unique keys mapping to one or more values;

wherein the second key-value family comprises a second set of unique keys, each key in the second set of unique keys mapping to at least one compressed value.

28. The system of claim 26, wherein at least one of the one or more first values comprises a key of the second key-value family.

29. The system of claim 26, wherein at least one of the one or more first values comprises information identifying a byte sequential portion of one of the one or more uncompressed values.

30. The system of claim 26, wherein the search mechanism is configured to use the search criterion to obtain the one or more first values by selecting a key from the first key-value family that equals the search criterion.

31. The system of claim 26:

wherein the key-value data repository comprises a cluster of a plurality of computing nodes;

wherein at least one key of the first key-value family is mastered by at least one node of the plurality nodes and at least one other key of the first key-value family is mastered by at least one other node of the plurality of nodes;

wherein each and every node of the cluster of nodes is configured to obtain values for any key of the first key-value family.

32. A system comprising:

a key-value data repository comprising a first key-value family, a second key-value family, and a third key-value family;

a search mechanism comprising one or more processors and configured to;

obtain a first search criterion and one or more second search criteria;

use the first search criterion and the second search criteria to obtain one or more first values from the first-key value family;

use the one or more first values to obtain one or more second values from the second key-value family;

use the one or more second values to obtain one or more compressed values from the third key-value family;

return one or more uncompressed portions of the one or more compressed values as search results.

33. The system of claim 32, wherein the search mechanism is further configured to:

uncompress the one or more compressed values to produce one or more uncompressed values;

use the one or more second values to identify the one or more uncompressed portions from the one or more uncompressed values.

34. The system of claim 33, wherein at least one of the one or more second values comprises information identifying a byte sequential portion of one of the one or more uncompressed values.

35. The system of claim 32, wherein the search mechanism is further configured to:

generate at least one of the second criteria using the first criterion and a search parameter obtained from a user.

36. The system of claim 32, wherein the search mechanism is configured to use the first search criterion and the second search criteria to obtain the one or more first values by selecting values of a key of the first key-value family that equals the first search criterion where the selected values satisfy the second search criteria.

37. The system of claim 32:

wherein the first key-value family comprises a first set of unique keys and a set of values to which the first set of unique keys are mapped;

wherein the second key-value family comprises a second set of unique keys and a set of values to which the second set of unique keys are mapped;

wherein the third key-value family comprises a third set of unique keys and a set of values to which the third set of unique keys are mapped;

wherein at least one value of each key in the first set of unique keys matches a key of the second set of unique keys;

wherein at least one value of each key in the second set of unique keys matches a key of the third set of unique keys.

38. The system of claim 32:

wherein the key-value data repository comprises a cluster of a plurality of computing nodes;

wherein at least one key of the first key-value family is mastered by at least one node of the plurality nodes and at least one other key of the first key-value family is mastered by at least one other node of the plurality of nodes;

wherein each and every node of the cluster of nodes is configured to obtain values for any key of the first key-value family.