BIG-FAST DATA CONNECTOR BETWEEN IN-MEMORY DATABASE SYSTEM AND DATA WAREHOUSE SYSTEM

• US 20140156586A1
• Filed: 01/31/2013
• Published: 06/05/2014
• Est. Priority Date: 12/04/2012
• Status: Active Grant

• Current Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Original Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Inventors: Black, Charles M., Keene, Christopher T., Lamb, Richard, Oglesby, Barrett T.
• Granted Patent: US 9,183,271 B2
• Time in Patent Office: Days
• US Class Current: 707/600
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/2255 Hash tables
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP

Abstract

Embodiments of the present invention include systems and methods for insuring better query consistency between at least two different databases, where one faster database has more up-to-date information than another slower database, and wherein updates are typically applied to the faster database first and then to the slower database. In embodiments, the systems and methods also insure that a query to the slower database is not performed until a set of one or more updates from the faster database have been applied to that slower database.

Claims

1. An in-memory database system that supports fast ingest data transfer between the in-memory database system and a data warehouse system, the in-memory database system comprising:

an in-memory database instance that receives and stores data;

a listener, communicatively coupled to the in-memory database instance, that, responsive to receiving notification that a data event has occurred in the in-memory database instance, selects a queue to which the data event will be sent and sends the data event to the selected queue to be sent to the data warehouse system; and

the queue system comprising a plurality of queues, each queue comprising a proxy table communicatively coupled to receive the data event from the listener and an associated write-behind listener that determines whether an update condition has been being satisfied for the queue and responsive to an update condition being satisfied, sends the data event to the data warehouse system.

2. The in-memory database system of claim 1 wherein the listener selects a queue by performing the steps comprising:

hashing a primary key of the data event to obtain a hash code; and

selecting the queue for the data event by taking a modulus of the hash code by the number of queues.

3. The in-memory database system of claim 1 wherein the in-memory database system is a distributed system comprising a plurality of in-memory database system instances and wherein:

one of the in-memory database system instances comprises the listener and the queue system; and

the listener is communicatively coupled to receive notifications from the other in-memory database system instances of the distributed system.

4. The in-memory database system of claim 3 wherein a plurality of the in-memory database system instances each comprises a listener and a queue system.

5. A computer-implement method for facilitating data transfer between an in-memory database system and a data warehouse system comprising:

receiving notification at a listener that a data event has been occurred at the in-memory database system;

identifying a queue to which the data event should be sent;

sending the data event to the identified queue to be sent to the data warehouse system; and

responsive to an update condition being satisfied, sending the data event to the data warehouse system.

6. The computer-implement method of claim 5 wherein the step of sending the data event to the identified queue to be sent to the data warehouse system comprises:

placing a copy of the data event into a proxy table; and

responsive to a microbatch listener, which is associated with the proxy table and receives notification that the data event has been entered into the proxy table, determining that a batch condition has been satisfied, writing a microbatch which includes the data event, to the data warehouse.

7. The computer-implement method of claim 6 wherein the batch condition is user-defined.

8. The computer-implement method of claim 5 wherein the step of identifying a queue to which the data event should be sent comprises:

hashing a primary key of the data event to obtain a hash code; and

selecting the queue for the data event by taking a modulus of the hash code by the number of queues.

9. The computer-implement method of claim 5 wherein the update condition is based upon:

time between updates, number of data events in the queue, or both.

10. The computer-implement method of claim 5 wherein the in-memory database system is part of a distributed system comprising a plurality of in-memory database system instances and wherein one of the in-memory database system instances comprises the listener and the queues and the listener is communicatively coupled to receive notifications from the other in-memory database system instances of the distributed system.

11. The computer-implement method of claim 5 wherein the in-memory database system is part of a distributed system comprising a plurality of in-memory database system instances and wherein at least some of the in-memory database system instances comprise a listener and one or more queues and wherein each listener is communicatively coupled to send data events to the one or more queues in the at least some of the in-memory database system instances that comprise one or more queues.

12. The computer-implement method of claim 5 wherein at least some of the one or more queues are nested to form queue tiers and wherein one or more additional factors are used to assign the data event to a nested queue.

13. A fast ingest system for facilitating data transfer between an in-memory database system and a data warehouse system, the fast ingest system comprising:

a listener that, responsive to receiving notification that a data event has occurred in the in-memory database system, selects a queue in a queue system comprising a plurality of queues to which the data event will be sent and sends the data event to the selected queue to be sent to the data warehouse system; and

the queue system comprising a plurality of queues, each a queue comprising a proxy table communicatively coupled to receive the data event from the listener and an associated write-behind listener that determines whether an update condition has been being satisfied for the queue and responsive to an update condition being satisfied, writes the data event to the data warehouse system.

14. The fast ingest system of claim 13 wherein a queue comprises a set of queues, each queue having at least one update condition for determining when to write data events to the data warehouse.

15. The fast ingest system of claim 14 wherein at least some of the queues in the set of queues are nested to form queue tiers.

16. The fast ingest system of claim 13 wherein the listener selects a queue by performing the steps comprising:

hashing a primary key of the data event to obtain a hash code; and

selecting a queue for the data event by taking a modulus of the hash code by the number of queues.

17. The fast ingest system of claim 13 wherein the in-memory database system is part of a distributed system comprising a plurality of in-memory database system instances and wherein:

one of the in-memory database system instances comprises the listener and the queue system; and

the listener is communicatively coupled to receive notifications from the other in-memory database system instances of the distributed system.

18. The fast ingest system of claim 17 wherein a plurality of the in-memory database system instances each comprises a listener and a queue system.

19. The fast ingest system of claim 18 wherein at least one of the queue systems in the plurality of in-memory database system instances comprises a plurality of queues.

20. The fast ingest system of claim 19 wherein at least one of the plurality of queues comprises nested queues.