PAGE FEED FOR EFFICIENT DATAFLOW BETWEEN DISTRIBUTED QUERY ENGINES

• US 20130238656A1
• Filed: 03/12/2012
• Published: 09/12/2013
• Est. Priority Date: 03/12/2012
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: entIT Software LLC (Open Text Corporation)
• Inventors: Chen, Qiming, Hsu, Meichun
• Granted Patent: US 9,836,513 B2
• Time in Patent Office: Days
• US Class Current: 707/770
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

Page feed for efficient dataflow between distributed query engines is disclosed. An example method comprises storing a query result relation by a producer query as pages in a query engine buffer pool, and then sending these pages to a distributed caching platform (DCP) as the data source of another query. The example method also includes reading the result relation by a consumer query from the page stored in the DCP.

Claims

1. A method of page feed for efficient dataflow between distributed query engines, comprising:

storing a query result relation by a producer query as a page in a query engine buffer pool;

sending the pages to a distributed caching platform (DCP) as a data source of another query;

reading the query result relation by a consumer query from the page stored in the DCP.

2. The method of claim 1, further comprising when requesting the page, first checking a local buffer pool for the page, then checking a cache if the requested page is not in the local buffer pool, then checking a disk if the requested page is not in the cache and the buffer pool.

3. The method of claim 2, further comprising pinning the page if the page is found in the local buffer pool or the cache.

4. The method of claim 2, further buffering the page using an inclusion model.

5. The method of claim 1, further comprising selecting page size to balance efficiency of access to the DCP and query processing.

6. The method of claim 1, further comprising externalizing buffered pages of the distributed QEs.

7. The method of claim 6, further comprising transferring intermediate query results among the distributed QEs directly using page-feed.

8. The method of claim 7, wherein using the page-feed preserves internal data format.

9. The method of claim 1, further comprising handling relation data communication at a storage level.

10. A system with page feed for efficient dataflow between distributed query engines (QEs), comprising:

a query engine buffer pool to store a query result relation as a page;

a distributed caching platform (DCP) for receiving the page as a data source of another query;

a client to read the result relation from the page stored in the DCP.

11. The system of claim 10, wherein the page structure preserves internal data format.

12. The system of claim 10, wherein the DCP includes a database buffer pool for low-latency access to large volumes of data by multiple nodes.

13. The system of claim 10, wherein the database buffer pool includes a local buffer pool and is further extended across multiple machines.

14. The system of claim 10, wherein buffered pages are stored in a unified hash table including key-value pairs to extend the buffer pool with Memcached.

15. The system of claim 14, where the pages are hash partitioned to separate portions of the unified hash table residing on separate nodes.

16. The system of claim 14, wherein a buffered page is mapped to the key-value pair with tag identifying the buffered page and serialized to a string key for Memcached access.

17. The system of claim 16, wherein binary content of the buffered page is treated as a value corresponding to a page key, and the value is passed to API functions of data transfer by an entry pointer plus the length.

18. A system with page feed for efficient dataflow between distributed query engines (QEs), comprising:

a distributed caching platform (DCP) to store a query result relation as a page;

a buffer manager to move pages from a query engine buffer pool to the DCP and retrieve the pages from the DCP.

19. The system of claim 18 wherein the buffered pages are sent to different physical DCP sites based on the hash value of the page key

20. The system of claim 19 wherein the buffered pages are placed in multiple memory nodes and are accessed through a unified interface.