Page feed for efficient dataflow between distributed query engines

• US 9,836,513 B2
• Filed: 03/12/2012
• Issued: 12/05/2017
• Est. Priority Date: 03/12/2012
• Status: Expired due to Fees

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: entIT Software LLC (Open Text Corporation)
• Inventors: Chen, Qiming, Hsu, Meichun
• Primary Examiner(s): Thomas, Ashish K
• Assistant Examiner(s): Mahmood, Rezwanul
• Application Number:
  US13/418,055
• Publication Number:
  US 20130238656A1
• Time in Patent Office: 2,094 Days
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

Page feed for efficient dataflow between distributed query engines is disclosed. An example method comprises storing a query result relation by a producer query as pages in a query engine buffer pool, and then sending these pages to a distributed caching platform (DCP) as the data source of another query. The example method also includes reading the result relation by a consumer query from the page stored in the DCP.

Claims

1. A method comprising:

storing a query result relation resulting from a producer query as a local cached page in a query processor buffer pool of a first query processor, wherein the query result relation comprises a plurality of tuples returned by the producer query, wherein the plurality of tuples comprise data values from a first column of a first database table, wherein the query processor buffer pool is to store a plurality of local cached pages, wherein each of the plurality of local cached pages is a fixed-length binary page identified by a local tag;

sending, using a binary protocol, the local cached page to a distributed caching platform (DCP) to be shared as an external cached page, wherein the external cached page is visible as a data source of another query, wherein the DCP is distributed over multiple nodes, wherein the external cached page is identified by an external key that includes different data from the local tag;

reading, using the binary protocol, the external cached page by a second query processor based on the external key; and

converting the external cached page to the plurality of tuples for use by a consumer query.

2. The method of claim 1, wherein the local tag includes a local identifier of the query result relation.

3. The method of claim 2, wherein the local tag further includes a table space identifier, a file identifier, and a sequence number of the local cached page.

4. The method of claim 2, wherein the external key includes a global identifier of the query result relation.

5. The method of claim 4, wherein the external key further includes a site identifier.

6. The method of claim 1, further comprising:

in response to sending the local cached page to the DCP;

converting the local tag to the external key, wherein the external key is included in a key-value pair; and

converting content of the local cached page to a value included in the key-value pair.

7. The method of claim 6, further comprising storing the key-value pair in a unified hash table residing on multiple nodes.

8. The method of claim 1, wherein reading the external cached page by a second query processor comprises using a plurality of master keys for the query result relation.

9. The method of claim 8, wherein the plurality of master keys is composed using a known list of site identifiers for a plurality of sites.

10. A system comprising:

a first query processor including a query processor buffer pool to store a query result relation as a local cached page, wherein each local cached page stored in the query processor buffer pool is of uniform size, wherein the query result relation comprises a plurality of tuples returned by a first query, wherein each local cached page is a fixed-length binary page identified by a local tag;

a distributed caching platform (DCP) to receive, using a binary protocol, the local cached page to be shared as an external cached page, wherein the external cached page is visible as a data source of another query, wherein the DCP comprises at least one hardware processor, wherein the DCP is distributed over multiple nodes, wherein the external cached page is identified by an external key that includes different data from the local tag;

a second query processor to read, using the binary protocol, the external cached page from the DCP based on the external key, and to convert the external cached page to the plurality of tuples for use by a consumer query.

11. The system of claim 10, wherein the local tag is composed from a local identifier of the first query result relation, a table space identifier, a file identifier, and a sequence number of the local cached page.

12. The system of claim 10, wherein the external key comprises a global identifier of the query result relation and a site identifier.

13. The system of claim 10, wherein each local cached page stored in the query processor buffer pool is a fixed-length binary page.

14. The system of claim 10, wherein the DCP is further to:

convert the local tag to the external key, wherein the external key is included in a key-value pair;

convert content of the local cached page to a value included in the key-value pair; and

store the key-value pair in a unified hash table.

15. The system of claim 14, where local cached pages of the query processor buffer pool are hash partitioned to separate portions of the unified hash table residing on separate nodes.

16. The system of claim 10, wherein the second query processor is to read the external cached page using a plurality of master keys of the query result relation.

17. The system of claim 16, wherein the plurality of master keys is composed using a known list of site identifiers for a plurality of sites.

18. A system comprising:

a first query processor including a query processor buffer pool to store a query result relation as a local cached page, wherein each local cached page stored in the query processor buffer pool is a fixed-length binary page identified by a local tag, wherein the query result relation comprises a plurality of tuples returned by a first query;

a distributed caching platform (DCP) to receive, using a binary protocol, the local cached page to be shared as an external cached page, wherein the external cached page is identified by an external key that includes different data from the local tag, wherein the DCP comprises at least one hardware processor;

a second query processor to read, using the binary protocol, the external cached page from the DCP based on the external key.

19. The system of claim 18 wherein the DCP is further to:

convert the local tag to the external key, wherein the external key is included in a key-value pair; and

convert content of the local cached page to a value included in the key-value pair.

20. The system of claim 19 wherein the DCP is further to store the key-value pair in a unified hash table.