Big-fast data connector between in-memory database system and data warehouse system

• US 9,934,263 B1
• Filed: 10/19/2015
• Issued: 04/03/2018
• Est. Priority Date: 12/04/2012
• Status: Active Grant

• Current Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Original Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Inventors: Black, Charles M., Keene, Christopher T., Lamb, Richard, Oglesby, Barrett T.
• Primary Examiner(s): Kuddus, Daniel
• Application Number:
  US14/886,640
• Time in Patent Office: 897 Days
• Field of Search: None
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

Embodiments of the present invention include systems and methods for insuring better query consistency between at least two different databases, where one faster database has more up-to-date information than another slower database, and wherein updates are typically applied to the faster database first and then to the slower database. In embodiments, the systems and methods also insure that a query to the slower database is not performed until a set of one or more updates from the faster database have been applied to that slower database.

Claims

1. A computer-implemented method comprising:

receiving, by a distributed in-memory database system in real-time or near real-time, a plurality of data events, wherein the in-memory database system comprises a plurality of in-memory database instances operating in parallel, and wherein a listener and a plurality of queues are installed on a single first in-memory database instance of the plurality of in-memory database instances, and wherein the listener receives, from one or more other in-memory database instances of the distributed in-memory database system, notifications of data events received by the one or more other in-memory database instances of the distributed in-memory database system;

receiving, by the listener, a notification that a data event has been received by one of the in-memory database instances of the distributed in-memory database system;

selecting a queue of the plurality of queues in which the data event should be stored, each queue of the plurality of queues comprising a respective microbatch listener;

storing the data event in the selected queue;

determining, by the microbatch listener of the selected queue, that an update condition has been satisfied for updating a data warehouse system with data events stored in the selected queue;

in response, storing the data event in the data warehouse system;

receiving, by the in-memory database system, a query;

obtaining first query results that satisfy the query from historical data in the data warehouse system, the historical data including the received data event stored in the data warehouse system;

obtaining second query results from the in-memory database instance that each identify in-memory data events that are not stored in the data warehouse system; and

providing the first query results and the second query results in a response to the query.

2. The method of claim 1, wherein one or more of the second query results identify data events not stored in the data warehouse system that were received by the in-memory database instance after the query was received.

3. The method of claim 1, wherein the update condition specifies a period of time, a number of received data events, or both.

4. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

receiving, by a distributed in-memory database system in real-time or near real-time, a plurality of data events, wherein the in-memory database system comprises a plurality of in-memory database instances operating in parallel, and wherein a listener and a plurality of queues are installed on a single first in-memory database instance of the plurality of in-memory database instances, and wherein the listener receives, from one or more other in-memory database instances of the distributed in-memory database system, notifications of data events received by the one or more other in-memory database instances of the distributed in-memory database system;

receiving, by the listener, a notification that a data event has been received by one of the in-memory database instances of the distributed in-memory database system;

selecting a queue of the plurality of queues in which the data event should be stored, each queue of the plurality of queues comprising a respective microbatch listener;

storing the data event in the selected queue;

determining, by the microbatch listener of the selected queue, that an update condition has been satisfied for updating a data warehouse system with data events stored in the selected queue;

in response, storing the data event in the data warehouse system;

receiving, by the in-memory database system, a query;

obtaining first query results that satisfy the query from historical data in the data warehouse system, the historical data including the received data event stored in the data warehouse system;

obtaining second query results from the in-memory database instance that each identify in-memory data events that are not stored in the data warehouse system; and

providing the first query results and the second query results in a response to the query.

5. The system of claim 4, wherein one or more of the second query results identify data events not stored in the data warehouse system that were received by the in-memory database instance after the query was received.

6. The system of claim 4, wherein the update condition specifies a period of time, a number of received data events, or both.

7. A computer program product comprising one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving, by a distributed in-memory database system in real-time or near real-time, a plurality of data events, wherein the in-memory database system comprises a plurality of in-memory database instances operating in parallel, and wherein a listener and a plurality of queues are installed on a single first in-memory database instance of the plurality of in-memory database instances, and wherein the listener receives, from one or more other in-memory database instances of the distributed in-memory database system, notifications of data events received by the one or more other in-memory database instances of the distributed in-memory database system;

receiving, by the listener, a notification that a data event has been received by one of the in-memory database instances of the distributed in-memory database system;

selecting a queue of the plurality of queues in which the data event should be stored, each queue of the plurality of queues comprising a respective microbatch listener;

storing the data event in the selected queue;

determining, by the microbatch listener of the selected queue, that an update condition has been satisfied for updating a data warehouse system with data events stored in the selected queue;

in response, storing the data event in the data warehouse system;

receiving, by the in-memory database system, a query;

obtaining first query results that satisfy the query from historical data in the data warehouse system, the historical data including the received data event stored in the data warehouse system;

obtaining second query results from the in-memory database instance that each identify in-memory data events that are not stored in the data warehouse system; and

providing the first query results and the second query results in a response to the query.

8. The computer program product of claim 7, wherein one or more of the second query results identify data events not stored in the data warehouse system that were received by the in-memory database instance after the query was received.

9. The computer program product of claim 7, wherein the update condition specifies a period of time, a number of received data events, or both.