Big-fast data connector between in-memory database system and data warehouse system

• US 9,183,271 B2
• Filed: 01/31/2013
• Issued: 11/10/2015
• Est. Priority Date: 12/04/2012
• Status: Active Grant

• Current Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Original Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Inventors: Black, Charles M., Keene, Christopher T., Lamb, Richard, Oglesby, Barrett T.
• Primary Examiner(s): KUDDUS, DANIEL A
• Application Number:
  US13/756,272
• Publication Number:
  US 20140156586A1
• Time in Patent Office: 1,013 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

Embodiments of the present invention include systems and methods for insuring better query consistency between at least two different databases, where one faster database has more up-to-date information than another slower database, and wherein updates are typically applied to the faster database first and then to the slower database. In embodiments, the systems and methods also insure that a query to the slower database is not performed until a set of one or more updates from the faster database have been applied to that slower database.

Claims

1. An in-memory database system that supports fast ingest data transfer between the in-memory database system and a data warehouse system, the in-memory database system comprising:

an in-memory database instance configured to receive data events in real-time or near real-time and update the in-memory database instance with the received data events;

a listener configured to receive notifications of data events from the in-memory database instance to select a queue of a plurality of queues in which the data event will be stored, and to store the data event in the selected queue; and

a plurality of queues, each queue comprising a proxy table and a corresponding write-behind listener, wherein each write-behind listener is configured to determine whether an update condition has been satisfied for a corresponding proxy table of the queue and, in response to determining that an update condition is satisfied, to store the data event in the data warehouse system,

and wherein the in-memory database system is configured to receive a query requesting query results that satisfy the query from both historical data in the data warehouse system and data events stored in the in-memory database instance that have not yet been stored in data warehouse system, to generate a time stamp for the query, to wait for the data warehouse system to be updated with the data event stored in the in-memory database instance that occurred before a time represented by the time stamp, and to generate query results that satisfy the query after the data warehouse system is updated with the data event stored in the in-memory database instance that occurred before the time represented by the time stamp for the query.

2. The in-memory database system of claim 1 wherein the listener is configured to select a queue by performing operations comprising:

hashing a primary key associated with the data event to obtain a hash code; and

generating a queue identifier for the data event by computing a modulus of the hash code and the number of queues.

3. The in-memory database system of claim 1 wherein the in-memory database system is a distributed system comprising a plurality of distinct in-memory database instances and wherein:

the listener and the queues are installed on a single in-memory database instance; and

the listener is configured to receive notifications of data events from other in-memory database instances of the distributed system.

4. The in-memory database system of claim 1, wherein the in-memory database system is a distributed system comprising a plurality of distinct in-memory database instances, and wherein a plurality of listeners are installed on a plurality of the in-memory database instances.

5. The system of claim 1, wherein the in-memory database system is configured to generate the query results by performing operations comprising:

obtaining first query results that satisfy the query from the historical data in the data warehouse; and

supplementing the first query results with second query results that satisfy the query from the data events in the in-memory database instance.

6. The system of claim 5, wherein supplementing the first query results with second query results that satisfy the query from the data events in the in-memory database instance comprises supplementing the first query results with data events received by the in-memory database instance after the query was received.

7. A computer-implemented method comprising:

receiving, by an in-memory database instance in real-time or near real-time, a plurality of data events;

receiving, by a listener, a notification that a data event has been received by the in-memory database instance;

selecting a queue of a plurality of queues in which the data event should be stored, each queue comprising a proxy table and a corresponding write-behind listener;

storing the data event in the selected queue;

receiving, by the in-memory database system, a query requesting query results that satisfy the query from both historical data in a data warehouse system and data events stored in the in-memory database instance that have not yet been stored in data warehouse system;

generating a time stamp for the query;

waiting for the data warehouse system to be updated with the data event stored in the in-memory database instance that occurred before a time represented by the time stamp;

determining, by a write-behind listener of the selected queue, that an update condition has been satisfied for updating the data warehouse system with data events stored in the selected queue;

in response, storing the data event in the data warehouse system;

and

generating query results that satisfy the query after the data warehouse system is updated with the data event stored in the in-memory database instance that occurred before the time represented by the time stamp for the query.

8. The computer-implemented method of claim 7, wherein selecting a queue comprises:

hashing a primary key associated with the data event to obtain a hash code; and

generating a queue identifier for the data event by computing a modulus of the hash code and the number of queues.

9. The computer-implemented method of claim 7, wherein the in-memory database system is a distributed system comprising a plurality of distinct in-memory database instances and wherein the listener and the queues are installed on a single in-memory database instance, and further comprising receiving, by the listener, notifications of data events from other in-memory database instances of the distributed system.

10. The computer-implemented method of claim 7, wherein the in-memory database system is a distributed system comprising a plurality of distinct in-memory database instances, and wherein a plurality of listeners are installed on a plurality of the in-memory database instances.

11. The method of claim 7, generating the query results comprises:

obtaining first query results that satisfy the query from the historical data in the data warehouse; and

supplementing the first query results with second query results that satisfy the query from the data events in the in-memory database instance.

12. The method of claim 11, wherein supplementing the first query results with second query results that satisfy the query from the data events in the in-memory database instance comprises supplementing the first query results with data events received by the in-memory database instance after the query was received.

13. A computer program product, encoded on one or more non-transitory computer storage media, comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

receiving, by an in-memory database instance in real-time or near real-time, a plurality of data events;

receiving, by a listener, a notification that a data event has been received by the in-memory database instance;

selecting a queue of a plurality of queues in which the data event should be stored, each queue comprising a proxy table and a corresponding write-behind listener;

storing the data event in the selected queue;

receiving, by the in-memory database system, a query requesting query results that satisfy the query from both historical data in a data warehouse system and data events stored in the in-memory database instance that have not yet been stored in data warehouse system;

generating a time stamp for the query;

waiting for the data warehouse system to be updated with the data event stored in the in-memory database instance that occurred before a time represented by the time stamp;

determining, by a write-behind listener of the selected queue, that an update condition has been satisfied for updating a data warehouse system with data events stored in the selected queue;

in response, storing the data event in the data warehouse system;

and

generating query results that satisfy the query after the data warehouse system is updated with the data event stored in the in-memory database instance that occurred before the time represented by the time stamp for the query.

14. The computer program product of claim 13, wherein selecting a queue comprises:

hashing a primary key associated with the data event to obtain a hash code; and

generating a queue identifier for the data event by computing a modulus of the hash code and the number of queues.

15. The computer program product of claim 13, wherein the in-memory database system is a distributed system comprising a plurality of distinct in-memory database instances and wherein the listener and the queues are installed on a single in-memory database instance, and further comprising receiving, by the listener, notifications of data events from other in-memory database instances of the distributed system.

16. The computer program product of claim 13, wherein the in-memory database system is a distributed system comprising a plurality of distinct in-memory database instances, and wherein a plurality of listeners are installed on a plurality of the in-memory database instances.

17. The computer program product of claim 13, generating the query results comprises:

obtaining first query results that satisfy the query from the historical data in the data warehouse; and

supplementing the first query results with second query results that satisfy the query from the data events in the in-memory database instance.

18. The computer program product of claim 17, wherein supplementing the first query results with second query results that satisfy the query from the data events in the in-memory database instance comprises supplementing the first query results with data events received by the in-memory database instance after the query was received.