AGGREGATION FRAMEWORK SYSTEM ARCHITECTURE AND METHOD

• US 20140032525A1
• Filed: 03/11/2013
• Published: 01/30/2014
• Est. Priority Date: 07/26/2012
• Status: Active Grant

• Current Assignee: MongoDB, Inc.
• Original Assignee: MongoDB, Inc.
• Inventors: Merriman, Dwight, Horowitz, Eliot
• Granted Patent: US 8,996,463 B2
• Time in Patent Office: Days
• US Class Current: 707/713
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/2453 Query optimisation
• CPC Class Codes: G06F 16/24549 Run-time optimisation
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: H04L 1/0076 Distributed coding, e.g. network coding, involving channel coding coding in both space and time H04L1/0618; cooperative diversity H04B7/022

Abstract

Database systems and methods that implement a data aggregation framework are provided. The framework can be configured to optimize aggregate operations over non-relational distributed databases, including, for example, data access, data retrieval, data writes, indexing, etc. Various embodiments are configured to aggregate multiple operations and/or commands, where the results (e.g., database documents and computations) captured from the distributed database are transformed as they pass through an aggregation operation. The aggregation operation can be defined as a pipeline which enables the results from a first operation to be redirected into the input of a subsequent operation, which output can be redirected into further subsequent operations. Computations may also be executed at each stage of the pipeline, where each result at each stage can be evaluated by the computation to return a result. Execution of the pipeline can be optimized based on data dependencies and re-ordering of the pipeline operations.

Claims

1. A computer implemented method for executing aggregation operations in a non-relational architecture, the method comprising the acts of:

receiving, at database routing system, database requests from computer executed processes;

routing, by the database routing system, the database requests to a plurality of database access managers over a communication network;

identifying, by an aggregation engine, an aggregation operation including a plurality of data operations;

determining, by the aggregation engine, an optimization for execution of the aggregation operation;

modifying, by the aggregation engine, the plurality of data operations to optimize execution; and

communicating, by the aggregation engine, the aggregation operation having at least one optimization for execution.

2. The method of claim 1, wherein the aggregation operation includes the plurality of data operations and a sequence of execution, and the act of modifying, by the aggregation engine, the plurality of data operations to optimize execution includes modifying the sequence of execution of the plurality of data operations.

3. The method of claim 1, wherein the method further comprises determining any data dependencies for a group of operations in the plurality of data operations.

4. The method of claim 3, further comprising an act of filtering a request for data defined by the group of operations based on the data dependencies.

5. The method of claim 3, further comprising an act of identifying a plurality of data fields accessed by the group of operations.

6. The method of claim 5, further comprising an act of limiting instantiation of any data field not needed for responding to the plurality of operations during execution.

7. The method of claim 5, further comprising generating data exclusion operations and introducing the data exclusion operations into the plurality of data operations.

8. The method of claim 1, further comprising an act of optimizing execution of the aggregation operation before routing the operations to the plurality of database access managers.

9. The method of claim 1, further comprising an act of optimizing execution of the aggregation operation after routing the operations to the plurality of database access managers.

10. The method of claim 1, further comprising an act of optimizing execution of the aggregation in association with routing and execution of the aggregation operation.

11. The method of claim 1, further comprising an act of identifying filtering operations with the plurality operations and alter an execution order of the plurality of operations to elevate execution of at least one filtering operation.

12. The method of claim 11, further comprising an act of identifying expansion operations with the plurality operations and combine execution of the expansion operation with at least one filtering operation.

13. The method of claim 1, further comprising managing, by the plurality of database access managers, data access to at least one partition of the data in the distributed database.

14. The method of claim 1, further comprising an act of organizing the at least one partition of the data according to a base unit of data storage having a non-relational structure.

15. The method of claim 1, wherein the base unit of storage includes a document, and the document comprises a data entity including zero or more key/value pairs

16. A distributed database system for optimizing aggregation operations in a non-relational architecture, the system comprising:

at least one processor operatively connected to a memory, the processor when executing is configured to execute a plurality of system components, wherein the plurality of system components include;

a router component configured to;

receive database requests from computer executed processes, and

route the database requests to a plurality of database access managers over a communication network;

an aggregation engine configured to;

identify an aggregation operation including a plurality of data operations,

determine an optimization for execution of the aggregation operation,

modify the plurality of data operations to optimize execution, and

communicate the aggregation operation having at least one optimization; and

wherein the plurality of database mangers are configured to execute the modified plurality of data operations to return data from the distributed database.

17. The distributed database system of claim 16, wherein the aggregation operation includes

the plurality of data operations and a sequence of execution, and wherein the aggregation engine is further configured to modify the sequence of execution of the plurality of data operations.

18. The distributed database system of claim 16, wherein the aggregation engine is further configured to determine any data dependencies for a group of operations in the plurality of data

operations.

19. The distributed database system of claim 18, wherein the aggregation engine is further configured to filter a request for data within the group of operations based on the data dependencies.

20. The distributed database system of claim 18, where the aggregation engine is further

configured to identify a plurality of data fields accessed by the group of operations.

21. The distributed database system of claim 20, wherein the aggregation engine is further

configured to limit instantiation of any data field not needed for responding to the plurality of operations during execution.

22. The distributed database system of claim 20, wherein the aggregation engine is further configured to generate data exclusion operations and introduce the data exclusion operations into

the plurality of data operations.

23. The distributed database system of claim 16, wherein the aggregation engine is further configured to optimize execution of the aggregation operation before routing the operations to the plurality of database access managers.

24. The distributed database system of claim 16, wherein the aggregation engine is further configured to optimize execution of the aggregation operation after routing the operations to the plurality of database access managers.

25. The distributed database system of claim 16, wherein the aggregation engine is further configured to optimize execution of the aggregation in association with routing and execution of the aggregation operation.

26. The distributed database system of claim 16, wherein the aggregation engine is further

configured to identify filtering operations with the plurality operations and alter an execution order of the plurality of operations to elevate execution of at least one filtering operation.

27. The distributed database system of claim 26, wherein the aggregation engine is further configured to identify expansion operations with the plurality operations and combine execution

of the expansion operation with at least one filtering operation.

28. The distributed database system of claim 16, wherein each of the plurality of database access managers manages data access to at least one partition of the data in the distributed

database.

29. The distributed database system of claim 16, wherein the at least one partition of the data comprises data organized according to a base unit of data storage.

30. The distributed database system of claim 16, wherein the base unit of storage includes a document, and the document comprises a data entity including zero or more key/value pairs.

31-85. -85. (canceled)