Document store with non-uniform memory access aware high performance query processing

• US 10,678,793 B2
• Filed: 11/17/2016
• Issued: 06/09/2020
• Est. Priority Date: 11/17/2016
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP SE
• Inventors: Geissinger, Steffen
• Primary Examiner(s): Featherstone, Mark D
• Assistant Examiner(s): Halm, Kweku William
• Application Number:
  US15/354,841
• Publication Number:
  US 20180137172A1
• Time in Patent Office: 1,300 Days
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

Systems and methods are described for implementation by a computing system having a non-uniform memory access (NUMA) architecture comprising a plurality of NUMA nodes. A query of a document store storing a collection of slices each comprising one or more documents is received from a client. Thereafter, it is determined which of the slices within the document store are required for execution of the query. An execution plan is then generated, using the query, that comprises a plurality of nodes each specifying at least one database operation to execute a portion of the query. The execution plan assigns one of the plurality of NUMA nodes to each slice determined to be required for execution of the query. The database operations specified by the nodes of the execution plan are then executed using the corresponding assigned NUMA nodes for the associated slice. Data responsive to the query is then provided to the client.

Claims

1. A method for implementation by a computing system having a non-uniform memory access (NUMA) architecture comprising a plurality of NUMA nodes, the method comprising:

receiving, from a client, a query of a document store storing a collection of slices each comprising one or more documents, wherein memory associated with each slice is assigned to one of the plurality of NUMA nodes;

determining which of the slices within the document store are required for execution of the query to form a list of slices;

generating, using the query, an execution plan comprising a plurality of nodes including a source node and a plurality of child nodes each specifying at least one database operation to execute a portion of the query, the execution plan comprising a plurality of pipelines of nodes, the execution plan assigning one of the plurality of NUMA nodes to each slice determined to be required for execution of the query, wherein each slice is aware of the NUMA node assigned thereto;

executing the database operations specified by the nodes of the execution plan using the corresponding assigned NUMA nodes for the associated slices, the executing comprising the source node going through the list of slices and forwarding each of the slices including the corresponding one or more documents as a data package to a corresponding child node of the execution plan; and

providing data responsive to the query to the client;

wherein;

each pipeline is assigned to a different one of the plurality of NUMA nodes; and

only one thread executes a node of the execution graph at any given time.

2. The method of claim 1, wherein the document store stores the documents in JavaScript Object Notation (JSON) format.

3. The method of claim 1, wherein the document store comprises a plurality of partitions defined by a partition specification.

4. The method of claim 3, wherein each partition comprises a plurality of slices and at least one index.

5. The method of claim 1, wherein the document store forms part of a database system having a second database having a different data storage format from the document store and wherein the query implicates data in the second database.

6. The method of claim 5 further comprising:

coordinating execution of a plurality of transactions between the document store and the second database.

7. The method of claim 6 further comprising:

persisting all changes specified by the query in the second database including changes implicating data stored within the document store.

8. The method of claim 7, wherein the second database is an in-memory relational database and the document store stores data on physical disk storage.

9. The method of claim 1, wherein each NUMA node comprises at least one associated data processor and memory storing instructions for execution by the associated at least one data processor.

10. A system comprising:

a document store storing a collection of slices each comprising one or more documents; and

a plurality of non-uniform memory access (NUMA) nodes each comprising at least one data processor and memory for storing instructions for execution by the associated at least one data processor;

wherein;

memory associated with each slice is assigned to one of the plurality of NUMA nodes;

the document store receives a query from a client;

it is determined which of the slices within the document store are required for execution of the query to form a list of slices;

an execution plan is generated, using the query, the execution plan comprising a plurality of nodes including a source node and a plurality of child nodes each specifying at least one database operation to execute a portion of the query, the execution plan comprising a plurality of pipelines of nodes, the execution plan assigning one of the plurality of NUMA nodes to each slice determined to be required for execution of the query, wherein each slice is aware of the NUMA node assigned thereto;

the database operations specified by the nodes of the execution plan are executed using the corresponding assigned NUMA nodes for the associated slices, the executing comprising the source node going through the list of slices and forwarding each of the slices including the corresponding one or more documents as a data package to a corresponding child node of the execution plan;

data responsive to the query is provided to the client;

each pipeline is assigned to a different one of the plurality of NUMA nodes; and

only one thread executes a node of the execution graph at any given time.

11. The system of claim 10, wherein the document store stores the documents in JavaScript Object Notation (JSON) format.

12. The system of claim 10, wherein the document store comprises a plurality of partitions defined by a partition specification.

13. The system of claim 12, wherein each partition comprises a plurality of slices and at least one index.

14. The system of claim 10, wherein the document store forms part of a database system having a second database having a different data storage format from the document store and wherein the query implicates data in the second database.

15. The system of claim 14, wherein execution of a plurality of transactions is coordinated between the document store and the second database.

16. The system of claim 15, wherein all changes specified by the query are persisted in the second database including changes implicating data stored within the document store.

17. The system of claim 16, wherein the second database is an in-memory relational database and the document store stores data on physical disk storage.

18. A non-transitory computer program product storing instructions for implementation by a computing system having a non-uniform memory access (NUMA) architecture comprising a plurality of NUMA nodes, the instructions, when executed, result in operations comprising:

receiving, from a client, a query of a document store storing a collection of slices each comprising one or more documents, wherein memory associated with each slice is assigned to one of the plurality of NUMA nodes;

determining which of the slices within the document store are required for execution of the query to form a list of slices;

generating, using the query, an execution plan comprising a plurality of nodes including a source node and a plurality of child nodes each specifying at least one database operation to execute a portion of the query, the execution plan comprising a plurality of pipelines of nodes, the execution plan assigning one of the plurality of NUMA nodes to each slice determined to be required for execution of the query, wherein each slice is aware of the NUMA node assigned thereto;

executing the database operations specified by the nodes of the execution plan using the corresponding assigned NUMA nodes for the associated slices, the executing comprising the source node going through the list of slices and forwarding each of the slices including the corresponding one or more documents as a data package to a corresponding child node of the execution plan;

providing data responsive to the query to the client;

each pipeline is assigned to a different one of the plurality of NUMA nodes;

only one thread executes a node of the execution graph at any given time;

for a node of the execution graph, internal data structures are initiated lazily when such node is first triggered;

a tick is called by a node of the execution graph for each new incoming data package; and

a finishing event is invoked by a node of the execution graph after all of its corresponding parent nodes of the execution graph have been executed.

19. The computer program product of claim 18, wherein:

the document store stores the documents in JavaScript Object Notation (JSON) format;

the document store comprises a plurality of partitions defined by a partition specification;

each partition comprises a plurality of slices and at least one index.

20. The computer program product of claim 18, wherein:

the document store forms part of a database system having a second database having a different data storage format from the document store and wherein the query implicates data in the second database; and

the operations further comprise;

coordinating execution of a plurality of transactions between the document store and the second database; and

persisting all changes specified by the query in the second database including changes implicating data stored within the document store.