HIGH PERFORMANCE BIG DATA COMPUTING SYSTEM AND PLATFORM

• US 20160299944A1
• Filed: 04/12/2016
• Published: 10/13/2016
• Est. Priority Date: 04/13/2015
• Status: Active Grant

• Current Assignee: Risk Management Solutions, Inc. (Moody's Corporation)
• Original Assignee: Risk Management Solutions, Inc. (Moody's Corporation)
• Inventors: Glaubman, David, Lynn, Dan, McCormick, Ian, Isaacson, Cory, Bornhvd, Christof
• Granted Patent: US 10,387,414 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/221 Column-oriented storage; Management thereof
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

A computing system and platform uses various types of data stores to allow efficient querying of, and accelerated access to, extremely large data sets. One such data store is a dimension store that combines key-value and columnar stores, access to which is provided by several selectable mechanisms chosen based the nature of the data of interest. These include bitmap-based access, us of an optimized columnar data format, and access via namespace identifiers. A compressed, optimized page data format is provided for storing and analyzing large fact-based data. The complex dimension store is used to provide complex relationships and interpretation of the fact-based data, enabling high-performance advanced queries, with bitmap indexes passed between the two stores. Dimension data is stored in an encrypted manner throughout the system, and can be exchanged among parties in a secure manner.

Claims

1. A computing system for accelerating large data transfer, the system including:

a first processor;

a first data store coupled to the first processor and configured as a dimension store to store input data elements;

a second processor;

a second data store coupled to the second processor and configured to store model data related to the input data elements;

a third processor operatively coupled with the first processor and the second processor; and

a third data store coupled to the third processor and configured to store analytics results, wherein the third processor is coupled to the first data store and the second data store to selectively retrieve portions of the input data elements and portions of the model data and to generate therefrom the analytics results, the third processor further configured to generate a query using characteristics of the input data and the model data to determine the portions of the input data elements and the portions of the model data to apply in the query of the analytics results.

2. A computing system as in claim 1, wherein the input data elements correspond to geographical data.

3. A computing system as in claim 1, wherein the query is configured to filter, aggregate or group data processed by the third processor.

4. A computing system as in claim 1, wherein the query uses a compressed bitmap index.

5. A computing system as in claim 1, wherein the first processor is further configured to compress or encrypt the portions of the input data elements.

6. A computing system as in claim 1, wherein the third processor is further configured to compress or encrypt results data generated by the third processor.

7. A computing system as in claim 1, wherein the third processor is configured to receive key information from the first data store and to use the key information to retrieve a subset of the portions of the analytics results data from the third data store.

8. A computing system as in claim 1, wherein the third processor is configured to retrieve, in an encrypted form, a subset of the analytics results data based on the portions of the input data elements and of the portions of the model data.

9. A computing system as in claim 1, wherein the third data store is configured to store data using OCDF.

10. A computing system as in claim 1, wherein the first data store is configured to maintain the input data according to arrays of columns.

11. A computing system as in claim 1, wherein the first data store is proximate to the third processor.

12. A computing system as in claim 1, wherein the third processor is configured to determine the portions of the input data based in part on data currently available in the third data store.

13. A computing system as in claim 1, wherein the third processor is configured to define data cubes to determine, responsive to the query, the portions of the input data elements, the portions of the model data and the portions of the analytics results data.

14. A computing system as in claim 1, wherein the second processor is configured to create models by combining and integrating functions using a Declarative Computation and Query Language.

15. A computing system as in claim 1, wherein the second processor is configured to invoke functions using a syntax corresponding to SQL.

16. A computing system as in claim 1, further comprising a platform services processor managing data streams to select among available processors to perform a plurality of processing tasks on a subset of data stored in at least one of the first data store, the second data store and the third data store.

17. A computing system as in claim 1, wherein the first processor is configured to transfer the portions of the input data elements in an encrypted form.

18. A method of moving data from first and second portions of a computing system to a third portion of a computing system, comprising:

generating a characteristic relating to the first and second portions;

generating a query relating to processing by the third portion; and

transferring subsets of data from the first and second portions to the third portion based on the query and on the characteristic.

19. The method of claim 18, wherein the characteristic is a compressed bitmap index.

20. The method of claim 18, wherein the first and second portions are compressed and encrypted, further generating the subsets in compressed and encrypted form, and wherein the transferring is of the subsets in the compressed and encrypted form.