Automatic generation of composite datasets based on hierarchical fields

• US 9,542,446 B1
• Filed: 01/14/2016
• Issued: 01/10/2017
• Est. Priority Date: 12/17/2015
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Woody, Patrick, Mehta, Rahul, Duffield, Ben
• Primary Examiner(s): ALAM, SHAHID AL
• Application Number:
  US14/996,179
• Time in Patent Office: 362 Days
• Field of Search: 707/701, 707/736, 707/754
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2428 Query predicate definition using graphical user interfaces, including menus and forms G06F16/2423 takes precedence
• CPC Class Codes: G06F 16/243 Natural language query formulation
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/2456 Join operations
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/90324 using system suggestions
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems

Abstract

Datasets are annotated with metadata including categories. Each category corresponds to one or more fields. A hierarchy mapping is generated to indicate a hierarchical relationship between different categories. A natural language query specifies a first granularity level indicating a particular category and one or more field values corresponding to the particular category. Based on the hierarchy mapping, one or more categories that are hierarchically related to the particular category are identified. Based on the metadata, two or more datasets that include at least one hierarchically related category is selected. Based on the first granularity level, one or more dataset filters are generated. The one or more dataset filters are translated to a second granularity level corresponding to the at least one hierarchically related category. The translated filters are applied to at least one of the selected datasets. The two or more datasets are joined to generate a composite dataset.

Claims

1. A method, comprising:

at one or more computing devices comprising one or more processors and memory storing one or more computer programs executed by the one or more processors to perform the method, performing the operations of;

annotating a plurality of datasets with metadata, the plurality of datasets including a plurality of fields, the metadata including a plurality of categories, each category of the plurality of categories corresponding to one or more fields of the plurality of fields;

generating a hierarchy mapping that indicates a hierarchical relationship between different categories of the plurality of categories;

receiving a natural language query that specifies a first granularity level that indicates a particular category of the plurality of categories and one or more field values that correspond to the particular category;

determining against which of the plurality of datasets to execute the natural language query based on;

identifying, based on the hierarchy mapping, one or more categories of the plurality of categories that are hierarchically related to the particular category;

selecting, based on the metadata, two or more datasets of the plurality of datasets that include at least one category of the one or more categories;

generating one or more dataset filters based on the first granularity level;

translating the one or more dataset filters to a second granularity level corresponding to the at least one category;

after translating the one or more dataset filters, applying the one or more dataset filters to at least one dataset of the two or more datasets;

joining the two or more datasets to generate a composite dataset that includes the particular category; and

executing the natural language query against the composite dataset to return a result for the natural language query.

2. The method of claim 1, wherein the metadata includes one or more measures, each measure of the one or more measures depending on at least one field of the plurality of fields.

3. The method of claim 1, wherein the natural language query specifies a particular measure.

4. The method of claim 1, wherein the natural language query is constructed based on one or more inputs received at a user interface comprising a plurality of predetermined query parameters that are selectable.

5. The method of claim 1, wherein the natural language query is constructed based on text input that is autocompleted.

6. The method of claim 1, wherein based on the metadata, selecting two or more datasets of the plurality of datasets that include at least one category of the one or more categories comprises:

ranking at least some of the plurality of datasets based on the hierarchical relationship between the particular category and the at least one category.

7. The method of claim 1, wherein based on the metadata, selecting two or more datasets of the plurality of datasets that include at least one category of the one or more categories comprises:

ranking at least some of the plurality of datasets based on a number of the one or more categories that are included in a particular dataset of the at least some of the plurality of datasets.

8. The method of claim 1, wherein the at least one dataset is translated to a level of the particular category and stored.

9. The method of claim 1, wherein the plurality of datasets includes a particular composite dataset that is generated based on all available data related to the particular category.

10. The method of claim 1, wherein the result is returned based on displaying the result as a line graph or a histogram.

11. A system, comprising:

storage media;

one or more processors;

and one or more programs stored in the storage media and configured for execution by the one or more processors, the one or more programs comprising instructions for;

annotating a plurality of datasets with metadata, the plurality of datasets including a plurality of fields, the metadata including a plurality of categories, each category of the plurality of categories corresponding to one or more fields of the plurality of fields;

generating a hierarchy mapping that indicates a hierarchical relationship between different categories of the plurality of categories;

receiving a natural language query that specifies a first granularity level that indicates a particular category of the plurality of categories and one or more field values that correspond to the particular category;

determining against which of the plurality of datasets to execute the natural language query based on;

identifying, based on the hierarchy mapping, one or more categories of the plurality of categories that are hierarchically related to the particular category;

selecting, based on the metadata, two or more datasets of the plurality of datasets that include at least one category of the one or more categories;

generating one or more dataset filters based on the first granularity level;

translating the one or more dataset filters to a second granularity level corresponding to the at least one category;

after translating the one or more dataset filters, applying the one or more dataset filters to at least one dataset of the two or more datasets;

joining the two or more datasets to generate a composite dataset that includes the particular category; and

executing the natural language query against the composite dataset to return a result for the natural language query.

12. The system of claim 11, wherein the metadata includes one or more measures, each measure of the one or more measures depending on at least one field of the plurality of fields.

13. The system of claim 11, wherein the natural language query specifies a particular measure.

14. The system of claim 11, wherein the natural language query is constructed based on one or more inputs received at a user interface comprising a plurality of predetermined query parameters that are selectable.

15. The system of claim 11, wherein the natural language query is constructed based on text input that is autocompleted.

16. The system of claim 11, wherein based on the metadata, selecting two or more datasets of the plurality of datasets that include at least one category of the one or more categories comprises:

ranking at least some of the plurality of datasets based on the hierarchical relationship between the particular category and the at least one category.

17. The system of claim 11, wherein based on the metadata, selecting two or more datasets of the plurality of datasets that include at least one category of the one or more categories comprises:

ranking at least some of the plurality of datasets based on a number of the one or more categories that are included in a particular dataset of the at least some of the plurality of datasets.

18. The system of claim 11, wherein the at least one dataset is translated to a level of the particular category and stored.

19. The system of claim 11, wherein the plurality of datasets includes a particular composite dataset that is generated based on all available data related to the particular category.

20. The system of claim 11, wherein the result is returned based on displaying the result as a line graph or a histogram.