Parallel query processing

• US 9,952,894 B1
• Filed: 01/27/2015
• Issued: 04/24/2018
• Est. Priority Date: 01/27/2014
• Status: Active Grant

• Current Assignee: Microstrategy Incorporated
• Original Assignee: Microstrategy Incorporated
• Inventors: Cappiello, Scott
• Primary Examiner(s): Lane, John A
• Application Number:
  US14/606,173
• Time in Patent Office: 1,183 Days
• CPC Class Codes: G06F 11/30 Monitoring
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3433 for load management allocation of a server based on load conditions G06F9/505; load rebalancing G06F9/5083; redistributing the load in a network by a load balancer H04L67/1029
• CPC Class Codes: G06F 2201/87 Monitoring of transactions
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system

Abstract

Systems and methods are disclosed for distributing an in-memory data store over a plurality of independent data partitions. For example, the method includes associating each of the plurality of independent data partitions with at least one of a plurality of processing units such that one or more data sets in a corresponding each of the plurality of independent data partitions are processed by the at least one of the plurality of processing units. A query execution engine is provided for causing the plurality of processing units to execute, in parallel, a series of queries to the plurality of independent data partitions.

Claims

1. A computer-implemented method comprising:

distributing an in-memory data store over a plurality of independent data partitions;

associating each of the plurality of independent data partitions with at least one of a plurality of processing units such that one or more data sets in one of the plurality of independent data partition are processed by a respective at least one of the plurality of processing units;

providing a query execution engine for causing the plurality of processing units to execute, in parallel, a series of queries to the plurality of independent data partitions;

receiving a transaction to be processed on the in-memory data store for an interface of a dashboard application;

determining that a visualization of the interface represents first data stored in a first independent data partition of the plurality of independent data partitions and second data stored in a second independent data partition of the plurality of independent data partitions;

based on the transaction and based on the first data being stored in the first independent data partition of the plurality of independent data partitions, generating a first query for accessing the first data stored in the first independent data partition of the plurality of independent data partitions;

based on the transaction and based on the second data being stored in the second independent data partition of the plurality of independent data partitions, generating a second query for accessing the second data stored in the second independent data partition of the plurality of independent data partitions; and

providing, for execution in parallel, the first query to the first independent data partition and the second query to the second independent data partition.

2. The computer-implemented method of claim 1 wherein one or more of the plurality of processing units comprise individual processor cores of a multi-core processor.

3. The computer-implemented method of claim 1, wherein the distributing the in-memory data store over the plurality of independent data partitions comprises partitioning one or more tables of the in-memory data store across the plurality of independent data partitions as the plurality of independent data partitions.

4. The computer-implemented method of claim 1, further comprising sharing one or more tables of the in-memory data store over the plurality of independent data partitions.

5. The computer-implemented method of claim 1, further comprising presenting, within the interface of the dashboard application, a representation of results from the processing of the first query and the second query.

6. The computer-implemented method of claim 5, further comprising:

providing, via the dashboard application, a user interface for receiving user input defining one or more filters related to first desired information; and

generating the first query and the second query in response to the user input defining the one or more filters related to the first desired information.

7. The computer-implemented method of claim 6, wherein the first desired information comprises market intelligence data.

8. The computer-implemented method of claim 7, further comprising loading into the in-memory data store all market intelligence data used by the dashboard application.

9. The computer-implemented method of claim 1, wherein a master node and a worker node comprise one or more of the plurality of processing units.

10. The computer-implemented method of claim 1, further comprising:

monitoring a processing status of the plurality of processing units; and

balancing a processing load across the plurality of processing units in response to the monitored processing status.

11. The computer-implemented method of claim 1, further comprising:

detecting one of a failure state or an abnormal condition state of the at least one of the plurality of processing units; and

in response to the detected state, effecting a failover to another one of the plurality of processing units to ensure continued processing of the first query and the second query.

12. The computer-implemented method of claim 1, further comprising, based on a user input, initiating the transaction at a selected one of a) an application programming interface layer, b) a dashboard engine, or c) both the application programming interface layer and the dashboard engine.

13. The computer-implemented method of claim 1, further comprising loading all data associated with a complete application into the in-memory data store.

14. The computer-implemented method of claim 1, further comprising loading two or more star schemas into the in-memory data store.

15. The computer-implemented method of claim 1, further comprising loading into the in-memory data store two or more fact tables.

16. The computer-implemented method of claim 15, wherein the fact tables comprise fact tables with varying levels of granularity.

17. The computer-implemented method of claim 1, further comprising loading into the in-memory data store one or more sets of many-to-many relationship tables.

18. A computer-implemented method comprising:

receiving a transaction to be carried out on an in-memory data store for an interface of a dashboard application;

determining that a visualization of the interface represents first data stored in a first independent data partition of the in-memory data store and second data stored in a second independent data partition of the in-memory data store;

based on the transaction and based on the first data stored in the first independent data partition of the in-memory data store, generating a first query for accessing the first independent data partition of the in-memory data store;

based on the transaction and based on the second data being stored in the second independent data partition of the in-memory data store, generating a second query for accessing the second data stored in the second independent data partition of the in-memory data store;

executing, in parallel, by a first processing unit associated with the first independent data partition, and by a second processing unit associated with the second independent data partition, the first query and the second query; and

updating, within the interface of the dashboard application, the visualization based on results from executing the first query and the second query in the in-memory data store.