Analysis of large bodies of textual data

• US 10,318,630 B1
• Filed: 08/16/2017
• Issued: 06/11/2019
• Est. Priority Date: 11/21/2016
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Kesin, Maxim, Gribelyuk, Paul
• Primary Examiner(s): Pullias, Jesse S
• Application Number:
  US15/678,874
• Time in Patent Office: 664 Days
• Field of Search: 704 1, 704 9, 704 10, 704257, 704270-275
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 40/137 Hierarchical processing, e.g. outlines
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking

Abstract

In various example embodiments, a textual identification system is configured to receive a set of search terms and identify a set of textual data based on the search terms. The textual identification system retrieves a data structure including textual identifications for the set of textual data and processes the data structure to generate a modified data structure. The textual identification system sums rows within the modified data structure and identifies one or more elements of interest. The textual identification system then causes presentation of the elements of interest in a first portion of a graphical user interface and the textual identifications for the set of textual data in a second portion of the graphical user interface.

Claims

1. A method, comprising:

receiving one or more search terms within a graphical user interface;

identifying, by one or more processors of a machine, a set of textual data based on the one or more search terms;

retrieving a data structure including textual identifications for the set of textual data and an indication of one or more data elements within one or more text sets of the set of textual data;

processing, by the one or more processors, the data structure to generate a modified data structure, the modified data structure generated by reducing to text sets included in the set of textual data identified based on the one or more search terms;

summing rows, by the one or more processors, within the modified data structure, the rows including values for data elements included in each of the identified set of textual data;

identifying, by the one or more processors, one or more elements of interest within the set of textual data based on the summed rows of the modified data structure;

determining, by the one or more processors, a context of occurrence for each element of interest;

normalizing, by the one or more processors, the elements of interest by removing redundant elements of interest based on the context of occurrence of two or more elements of interest and generating a normalized set of elements of interest; and

causing presentation of the normalized set of elements of interest in a first portion of the graphical user interface and the textual identifications for the set of textual data in a second portion of the graphical user interface.

2. The method of claim 1, wherein causing presentation of the elements of interest and the textual identifications further comprises:

causing presentation of at least a portion of a text set of the set of textual data in a third portion of the graphical user interface.

3. The method of claim 1, wherein causing presentation of the elements of interest further comprises:

identifying an element type for each of the elements of interest; and

causing presentation of a visual indicator differentiating the elements of interest based on an element type.

4. The method of claim 1, further comprising:

in response to determining the context of occurrence for each element of interest, generating a set of tokens for each element of interest, the set of tokens representing the context of occurrence;

identifying an overlap of two or more elements of interest based on the set of tokens for the two or more elements of interest; and

linking two or more elements of interest.

5. The method of claim 1 further comprising:

receiving a selection of a textual corpus from a set of textual corpora, each textual corpus of the set of textual corpora containing one or more text sets, the set of textual corpora identified from the selected textual corpus.

6. The method of claim 1, wherein identifying the set of textual data further comprises:

accessing a set of textual corpora, each textual corpus of the set of textual corpora containing one or more text sets; and

dynamically partitioning the set of textual corpora to identify a textual corpus from the set of textual corpora containing the set of text sets associated with the one or more search terms.

7. A computer implemented system, comprising:

one or more processors; and

a processor-readable storage device comprising processor-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving one or more search terms within a graphical user interface;

identifying a set of textual data based on the one or more search terms;

retrieving a data structure including textual identifications for the set of textual data and an indication of one or more data elements within one or more text sets of the set of textual data;

processing the data structure to generate a modified data structure, the modified data structure generated by reducing to text sets included in the set of textual data identified based on the one or more search terms;

summing rows within the modified data structure, the rows including values for data elements included in each of the identified set of textual data;

identifying one or more elements of interest within the set of textual data based on the summed rows of the modified data structure;

determining a context of occurrence for each element of interest;

normalizing the elements of interest by removing redundant elements of interest based on the context of occurrence of two or more elements of interest and generating a normalized set of elements of interest; and

causing presentation of the normalized set of elements of interest in a first portion of the graphical user interface and the textual identifications for the set of textual data in a second portion of the graphical user interface.

8. The system of claim 7, wherein causing presentation of the elements of interest and the textual identifications further comprises:

causing presentation of at least a portion of a text set of the set of textual data in a third portion of the graphical user interface.

9. The system of claim 7, wherein causing presentation of the elements of interest further comprises:

identifying an element type for each of the elements of interest; and

causing presentation of a visual indicator differentiating the elements of interest based on an element type.

10. The system of claim 7, wherein the operations further comprise:

in response to determining the context of occurrence for each element of interest, generating a set of tokens for each element of interest, the set of tokens representing the context of occurrence;

identifying an overlap of two or more elements of interest based on the set of tokens for the two or more elements of interest; and

linking two or more elements of interest.

11. The system of claim 7, wherein the operations further comprise:

receiving a selection of a textual corpus from a set of textual corpora, each textual corpus of the set of textual corpora containing one or more text sets, the set of textual corpora identified from the selected textual corpus.

12. The system of claim 7, wherein identifying the set of textual data further comprises:

accessing a set of textual corpora, each textual corpus of the set of textual corpora containing one or more text sets; and

dynamically partitioning the set of textual corpora to identify a textual corpus from the set of textual corpora containing the set of text sets associated with the one or more search terms.

13. A processor-readable storage device comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving one or more search terms within a graphical user interface;

identifying a set of textual data based on the one or more search terms;

retrieving a data structure including textual identifications for the set of textual data and an indication of one or more data elements within one or more text sets of the set of textual data;

processing the data structure to generate a modified data structure, the modified data structure generated by reducing to text sets included in the set of textual data identified based on the one or more search terms;

summing rows within the modified data structure, the rows including values for data elements included in each of the identified set of textual data;

identifying one or more elements of interest within the set of textual data based on the summed rows of the modified data structure;

determining a context of occurrence for each element of interest normalizing the elements of interest by removing redundant elements of interest based on the context of occurrence of two or more elements of interest and generating a normalized set of elements of interest; and

causing presentation of the normalized set of elements of interest in a first portion of the graphical user interface and the textual identifications for the set of textual data in a second portion of the graphical user interface.

14. The processor-readable storage device of claim 13, wherein causing presentation of the elements of interest and the textual identifications further comprises:

causing presentation of at least a portion of a text set of the set of textual data in a third portion of the graphical user interface.

15. The processor-readable storage device of claim 13, wherein causing presentation of the elements of interest further comprises:

identifying an element type for each of the elements of interest; and

causing presentation of a visual indicator differentiating the elements of interest based on an element type.

16. The processor-readable storage device of claim 13, wherein the operations further comprise:

in response to determining the context of occurrence for each element of interest, generating a set of tokens for each element of interest, the set of tokens representing the context of occurrence;

identifying an overlap of two or more elements of interest based on the set of tokens for the two or more elements of interest; and

linking two or more elements of interest.

17. The processor-readable storage device of claim 13, wherein identifying the set of textual data further comprises:

accessing a set of textual corpora, each textual corpus of the set of textual corpora containing one or more text sets; and

dynamically partitioning the set of textual corpora to identify a textual corpus from the set of textual corpora containing the set of text sets associated with the one or more search terms.