Organizing datasets for adaptive responses to queries

• US 10,592,508 B2
• Filed: 09/13/2017
• Issued: 03/17/2020
• Est. Priority Date: 09/13/2016
• Status: Active Grant

• Current Assignee: The Bank of New York Mellon (Bank of New York Mellon Corporation)
• Original Assignee: The Bank of New York Mellon (Bank of New York Mellon Corporation)
• Inventors: Srivastava, Kumar S.
• Primary Examiner(s): Spieler, William
• Application Number:
  US15/703,823
• Publication Number:
  US 20180075095A1
• Time in Patent Office: 916 Days
• CPC Class Codes: G06F 16/213 with details for schema evolution support
• CPC Class Codes: G06F 16/219 Managing data history or versioning querying versioned data G06F16/2474; querying temporal data G06F16/2477
• CPC Class Codes: G06F 16/2358 Change logging, detection, and notification replication G06F16/27
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/90324 using system suggestions

Abstract

A dataset management system organizes datasets and tracks the changes to the dataset to adaptively respond to user queries. For a dataset, the dataset management system tracks the evolving schema of the dataset over time as new data values and/or updates to existing data values are incorporated into the dataset. When a query is received, the dataset management system accesses the schema of a dataset to understand how the dataset has changed over time. Given the changing schema of the dataset, the dataset management system can respond by providing recommendations as to suggested queries that can return improved results. As another option, the dataset management system can execute a query and return results that satisfy the query to the client device that provided the query.

Claims

1. A method comprising:

receiving new data values for storage with a dataset;

for each new data value, extracting attributes of the new data value, at least one attribute describing a pattern in the new data value;

identifying a schema for the dataset based on the determined attributes of the new data values;

adding the identified schema to a dataset lineage map describing a set of schemas for the dataset;

receiving a query from a client device for data values of the dataset, the query specifying a data value type; and

validating the query against the dataset lineage map, the validation comprising;

determining whether the dataset includes the data value type specified by the query by accessing the identified schema for the dataset.

2. The method of claim 1, further comprising:

combining the received dataset with an existing dataset; and

identifying schema changes between the identified schema involving the new data values and a schema of an existing dataset.

3. The method of claim 2, wherein adding the identified schema to a dataset lineage map further comprises adding the identified schema changes between the identified schema and the schema of the existing dataset to the dataset lineage map.

4. The method of claim 1, wherein each schema in the set described by the dataset lineage map is associated with a time range for which the schema represents a current version of the dataset.

5. The method of claim 1, further comprising:

responsive to determining that the dataset includes the data value type specified by the query, generating results for the query; and

providing the generated results to the client device.

6. The method of claim 5, wherein generating results for the query comprises:

identifying one or more schemas described by the dataset lineage map that each correspond to a time range that is within a time range specified by the query; and

generating results for the query from datasets corresponding to the identified one or more schemas.

7. The method of claim 6, wherein generating results for the query further comprises:

parallel processing the datasets corresponding to the identified one or more schemas to generate individual results for each dataset; and

combining the individual results for each dataset to generate results for the query.

8. The method of claim 1, further comprising:

responsive to determining that the dataset does not include the data value type specified by the query, providing a suggested query to the client device.

9. The method of claim 8, wherein providing a suggested query to the client device further comprises:

identifying an additional schema in the set described by the dataset lineage map that includes the data type specified by the query; and

generating a suggested query comprising a recommended criteria, wherein the additional schema satisfies the recommended criteria of the suggested query and the identified schema for the dataset does not satisfy the recommended criteria of the suggested query.

10. The method of claim 9, wherein the query from the client device specifies a first date range and wherein the recommended criteria specifies a second date range, wherein the second date range is narrower than the first date range.

11. The method of claim 9, further comprising:

receiving an updated query from the client device, the updated query specifying the recommended criteria previously provided to the client device in the suggested query.

12. A non-transitory computer-readable storage medium comprising computer code that, when executed by a processor, causes the processor to:

receive new data values for storage with a dataset;

for each new data value, extract attributes of the new data value, at least one attribute describing a pattern in the new data value;

identify a schema for the dataset based on the determined attributes of the new data values;

add the identified schema to a dataset lineage map describing a set of schemas for the dataset;

receive a query from a client device for data values of the dataset, the query specifying a data value type; and

validate the query against the dataset lineage map, wherein the computer code that causes the processor to validate the query further comprises computer code that, when executed by the processor, causes the processor to;

determine whether the dataset includes the data value type specified by the query by accessing the identified schema for the dataset.

13. The non-transitory computer-readable storage medium of claim 12, further comprising computer code that, when executed, causes the processor to:

combine the received dataset with an existing dataset; and

identify schema changes between the identified schema involving the new data values and a schema of an existing dataset.

14. The non-transitory computer-readable storage medium of claim 13, wherein the computer code that causes the processor to add the identified schema to a dataset lineage map further comprises computer code that, when executed, causes the processor to:

add the identified schema changes between the identified schema and the schema of the existing dataset to the dataset lineage map.

15. The non-transitory computer-readable storage medium of claim 12, wherein each schema in the set described by the dataset lineage map is associated with a time range for which the schema represents a current version of the dataset.

16. The non-transitory computer-readable storage medium of claim 12, further comprising computer code that, when executed by the processor, causes the processor to:

responsive to the determination that the dataset includes the data value type specified by the query, generate results for the query; and

provide the generated results to the client device.

17. The non-transitory computer-readable storage medium of claim 16, wherein the computer code that causes the processor to generate results for the query further comprises computer code that, when executed by the processor, causes the processor to:

identify one or more schemas described by the dataset lineage map that each correspond to a time range that is within a time range specified by the query; and

generate results for the query from datasets corresponding to the identified one or more schemas.

18. The non-transitory computer-readable storage medium of claim 17, wherein the computer code that causes the processor to generate results for the query further comprises computer code that, when executed by the processor, causes the processor to:

parallel process the datasets corresponding to the identified one or more schemas to generate individual results for each dataset; and

combine the individual results for each dataset to generate results for the query.

19. The non-transitory computer-readable storage medium of claim 12, further comprising computer code that, when executed by the processor, causes the processor to:

responsive to the determination that the dataset does not include the data value type specified by the query, provide a suggested query to the client device.

20. The non-transitory computer-readable storage medium of claim 19, wherein the computer code that causes the processor to provide a suggested query to the client device further comprises computer code that, when executed by the processor, causes the processor to:

identify an additional schema in the set described by the dataset lineage map that includes the data type specified by the query; and

generate a suggested query comprising a recommended criteria, wherein the additional schema satisfies the recommended criteria of the suggested query and the identified schema for the dataset does not satisfy the recommended criteria of the suggested query.