Selecting backing stores based on data request

• US 10,437,807 B1
• Filed: 10/09/2017
• Issued: 10/08/2019
• Est. Priority Date: 07/06/2017
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Brockington-Hill, Christopher, Rickards, Neil
• Primary Examiner(s): Bowen, Richard L
• Application Number:
  US15/728,439
• Time in Patent Office: 729 Days
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/278 Data partitioning, e.g. horizontal or vertical partitioning
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database

Abstract

Techniques for improving database searches are described herein. In an embodiment, a server computer system stores one or more first datasets in a first data repository and one or more second datasets in a second data repository. The server computer receives a request to perform an analysis on a particular dataset. The server computer determines that the particular dataset is stored in the first data repository and the second data repository. Based, at least in part, on an attribute of the request, the server computer selects the second data repository and responds to the request with data from the particular dataset stored in the second data repository.

Claims

1. A data processing method comprising:

storing, in a columnar datastore, one or more first datasets;

storing, in an index data repository, one or more second datasets;

receiving a request to perform an analysis on a particular dataset;

in response to receiving the request to perform an analysis on the particular dataset, determining that the particular dataset is stored in the columnar datastore and an index of the particular dataset is stored in the index data repository, wherein the index data repository comprises one or more access control systems which provide access controls for the particular dataset;

in response to determining that the particular dataset is stored in the columnar datastore and an index of the particular dataset is stored in the index data repository, performing;

determining whether the request is at least partially for data that is subject to access controls;

if the request is at least partially for data that is subject to access controls, selecting the index data repository instead of the columnar datastore in response to a determination that the request is at least partially for data that is subject to access control and responding to the request with data from the index of the particular dataset stored in the index data repository;

if the request is not at least partially for data that is subject to access controls, determining whether to select the index data repository or the columnar datastore and responding to the request with data from the columnar datastore if the columnar datastore is selected.

2. The method of claim 1, further comprising:

receiving a second request for data from a second particular dataset;

determining that the second particular dataset is stored in the columnar datastore and an index of the second particular dataset is stored in the index data repository;

determining that the second request comprises a filter of the second particular dataset for one or more key terms;

selecting the index data repository based, at least in part, on the determination that the request comprises a filter of the second particular dataset for one or more key terms;

responding to the second request with data from the index of the second particular dataset stored in the index data repository.

3. The method of claim 1, further comprising:

receiving a second request for data from a second particular dataset;

determining that the second particular dataset is stored in the columnar datastore and an index of the second particular dataset is stored in the index data repository;

determining that the second request is at least partially for data from the second particular dataset which comprises less than a threshold number of rows of data;

selecting the columnar datastore store based, at least in part, on the determination that the second request is at least partially for data from the second particular dataset which comprises less than the threshold number of rows of data;

responding to the second request with data from the columnar datastore.

4. The method of claim 1, further comprising:

receiving a second request for data from a second particular dataset;

determining that the second particular dataset is stored in the columnar datastore and an index of the second particular dataset is stored in the index data repository;

determining that the second request is at least partially to perform an aggregation of values in one or more columns in the second particular dataset;

selecting the columnar datastore based, at least in part, on the determination that the request is at least partially to perform an aggregation of values in one or more columns of the second particular dataset;

responding to the request with data of the second particular dataset stored in the columnar datastore.

5. The method of claim 1, further comprising:

storing an access threshold value;

determining that, within a particular period of time, the particular dataset has been accessed from the columnar datastore a number of times that is greater than the access threshold value and, in response, storing an index of the particular dataset in the index data repository.

6. The method of claim 1, further comprising:

storing an attribute type threshold value;

determining that, within a particular period of time, a query with a particular type of attribute has been run against the particular dataset from the columnar datastore a number of times that is greater than the attribute type threshold value and, in response, storing an index of the particular dataset in the index data repository.

7. A system comprising:

one or more processors;

one or more storage media;

one or more instructions stored in the storage media which, when executed by the one or more processors, cause performance of;

storing, in a columnar datastore, one or more first datasets;

storing, in an index data repository, one or more second datasets;

receiving a request to perform an analysis on a particular dataset;

in response to receiving the request to perform an analysis on the particular dataset, determining that the particular dataset is stored in the columnar datastore and an index of the particular dataset is stored in the index data repository, wherein the index data repository comprises one or more access control systems which provide access controls for the particular dataset;

in response to determining that the particular dataset is stored in the columnar datastore and an index of the particular dataset is stored in the index data repository, performing;

determining whether the request is at least partially for data that is subject to access controls;

if the request is at least partially for data that is subject to access controls, selecting the index data repository instead of the columnar datastore in response to a determination that the request is at least partially for data that is subject to access control and responding to the request with data from the index of the particular dataset stored in the index data repository;

if the request is not at least partially for data that is subject to access controls, determining whether to select the index data repository or the columnar datastore and responding to the request with data from the columnar datastore if the columnar datastore is selected.

8. The system of claim 7, wherein the instructions, when executed by the one or more processors, further cause performance of:

receiving a second request for data from a second particular dataset;

determining that the second particular dataset is stored in the columnar datastore and an index of the second particular dataset is stored in the index data repository;

determining that the second request comprises a filter of the second particular dataset for one or more key terms;

selecting the index data repository based, at least in part, on the determination that the request comprises a filter of the second particular dataset for one or more key terms;

responding to the second request with data from the index of the second particular dataset stored in the index data repository.

9. The system of claim 7, wherein the instructions, when executed by the one or more processors, further cause performance of:

receiving a second request for data from a second particular dataset;

determining that the second particular dataset is stored in the columnar datastore and an index of the second particular dataset is stored in the index data repository;

determining that the second request is at least partially for data from the second particular dataset which comprises less than a threshold number of rows of data;

selecting the columnar datastore based, at least in part, on the determination that the second request is at least partially for data from the second particular dataset which comprises less than the threshold number of rows of data;

responding to the second request with data from the columnar datastore.

10. The system of claim 7, wherein the instructions, when executed by the one or more processors, further cause performance of:

receiving a second request for data from a second particular dataset;

determining that the second particular dataset is stored in the columnar datastore and an index of the second particular dataset is stored in the index data repository;

determining that the second request is at least partially to perform an aggregation of values in one or more columns in the second particular dataset;

selecting the columnar datastore based, at least in part, on the determination that the request is at least partially to perform an aggregation of values in one or more columns of the second particular dataset;

responding to the request with data of the second particular dataset stored in the columnar datastore.

11. The system of claim 7, wherein the instructions, when executed by the one or more processors, further cause performance of:

storing an access threshold value;

determining that, within a particular period of time, the particular dataset has been accessed from the columnar datastore a number of times that is greater than the access threshold value and, in response, storing an index of the particular dataset in the index data repository.

12. The system of claim 7, wherein the instructions, when executed by the one or more processors, further cause performance of:

storing an attribute type threshold value;

determining that, within a particular period of time, a query with a particular type of attribute has been run against the particular dataset from the columnar datastore a number of times that is greater than the attribute type threshold value and, in response, storing an index of the particular dataset in the index data repository.