High performance big data computing system and platform

• US 10,387,414 B2
• Filed: 04/12/2016
• Issued: 08/20/2019
• Est. Priority Date: 04/13/2015
• Status: Active Grant

• Current Assignee: Risk Management Solutions, Inc. (Moody's Corporation)
• Original Assignee: Risk Management Solutions, Inc. (Moody's Corporation)
• Inventors: Isaacson, Cory, Glaubman, David, Lynn, Dan, Bornhoevd, Christof, McCormick, Ian
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): Mueller, Kurt A
• Application Number:
  US15/097,233
• Publication Number:
  US 20160299944A1
• Time in Patent Office: 1,225 Days
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

A computing system and platform uses various types of data stores to allow efficient querying of, and accelerated access to, extremely large data sets. One such data store is a dimension store that combines key-value and columnar stores, access to which is provided by several selectable mechanisms chosen based the nature of the data of interest. These include bitmap-based access, us of an optimized columnar data format, and access via namespace identifiers. A compressed, optimized page data format is provided for storing and analyzing large fact-based data. The complex dimension store is used to provide complex relationships and interpretation of the fact-based data, enabling high-performance advanced queries, with bitmap indexes passed between the two stores. Dimension data is stored in an encrypted manner throughout the system, and can be exchanged among parties in a secure manner.

Claims

1. A computing system for accelerating large data transfer, the system including:

a first processor;

a dimension store coupled to the first processor and storing input data elements, the dimension store comprising a key-value store and a column store, wherein the key-value store stores the input data elements in a key-value data format, and wherein the column store stores the input data elements in a columnar data format;

a second processor;

a model data store coupled to the second processor and storing model data related to the input data elements;

a third processor coupled to the dimension store and the model data store, wherein the third processor is configured to, in response to receiving an analytics query;

determine that the key-value store can serve a first portion of the analytics query with better performance than the column store, and that the column store can serve a second portion of the analytics query with better performance than the key-value store;

in response to the determination, selectively retrieve, based on the first portion of the analytics query and second portion of the analytics query, a first portion of the input data elements from the key-value store and a second portion of the input data elements from the column store;

join the first portion of the input data elements and the second portion of the input data elements to generate a set of input data elements;

retrieve, based on the analytics query, portions of the model data; and

generate a set of analytics results based on the set of input data elements and the portions of the model data and store the set of analytics results in an analytics results store.

2. A computing system as in claim 1, wherein the input data elements correspond to geographical data.

3. A computing system as in claim 1, wherein the analytics query is configured to filter, aggregate or group data processed by the third processor.

4. A computing system as in claim 1, wherein the analytics query uses a compressed bitmap index.

5. A computing system as in claim 1, wherein the first processor is further configured to compress or encrypt the portions of the input data elements.

6. A computing system as in claim 1, wherein the third processor is further configured to compress or encrypt results data generated by the third processor.

7. A computing system as in claim 1, wherein the third processor is configured to receive key information from the dimension store and to use the key information to retrieve a subset of analytics results data from the analytics results store.

8. A computing system as in claim 1, wherein the third processor is configured to retrieve, in an encrypted form, a subset of analytics results data from the analytics results store, based on the first portion of the input data elements, the second portion of the input data elements and the portions of the model data.

9. A computing system as in claim 1, wherein the analytics results store is configured to store data using OCDF.

10. A computing system as in claim 1, wherein the column store is configured to maintain the input data elements according to arrays of columns.

11. A computing system as in claim 1, wherein the dimension store is proximate to the third processor.

12. A computing system as in claim 1, wherein the third processor is configured to determine the first and second portions of the input data elements based in part on data currently available in the analytics results store.

13. A computing system as in claim 1, wherein the third processor is configured to define data cubes to determine, responsive to the analytics query, the first portion of the input data elements, the second portion of the input data elements, and the portions of the model data.

14. A computing system as in claim 1, wherein the second processor is configured to create models by combining and integrating functions using a Declarative Computation and Query Language.

15. A computing system as in claim 1, wherein the second processor is configured to invoke functions using a syntax corresponding to SQL.

16. A computing system as in claim 1, further comprising a platform services processor managing data streams to select among available processors to perform a plurality of processing tasks on a subset of data stored in at least one of the dimension store, the model data store and the analytics results store.

17. A computing system as in claim 1, wherein the first processor is configured to transfer the portions of the input data elements in an encrypted form.

18. A method of moving data from first and second portions of a computing system to a third portion of a computing system, comprising:

receiving an analytics query and, based on the analytics query, generating a characteristic relating to the first and second portions of the computing system;

determining that the first portion of the computing system can serve a first portion of the analytics query with better performance than the second portion of the computing system, and that the second portion of the computing system can serve a second portion of the analytics query with better performance than the first portion of the computing system;

in response to the determining, retrieving a first subset of data from the first portion of the computing system and a second subset of data from the second portion of the computing system, wherein the first portion comprises a key-value data store, wherein the second portion comprises a column data store, wherein the key-value data store and the column data store each store input data elements, wherein the key-value data store stores the input data elements in a key-value data format, and wherein the column data store stores input data elements in a columnar data format;

transferring the first subset of data and the second subset of data from the first and second portions to the third portion based on the query and on the characteristic.

19. The method of claim 18, wherein the characteristic is a compressed bitmap index.

20. The method of claim 18, wherein the first and second portions are compressed and encrypted, further comprising generating the subsets in compressed and encrypted form, and wherein the transferring is of the subsets in the compressed and encrypted form.