APPARATUS AND METHOD FOR INTEGRATING MAP-REDUCE INTO A DISTRIBUTED RELATIONAL DATABASE

• US 20100257198A1
• Filed: 04/02/2009
• Published: 10/07/2010
• Est. Priority Date: 04/02/2009
• Status: Active Grant

• Current Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Original Assignee: Greenplum, Inc. (Dell Technologies Inc.)
• Inventors: Welton, Caleb E., Cohen, Jeffrey Ira, Lonergan, Luke
• Granted Patent: US 8,713,038 B2
• Time in Patent Office: Days
• US Class Current: 707/770
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/93 Document management systems

Abstract

A computer readable storage medium includes executable instructions to define a map-reduce document that coordinates processing of data in a distributed database. The map-reduce document complies with a map-reduce specification that integrates map-reduce functions with queries in a query language. The operations specified by the map-reduce document are executed in the distributed database.

Claims

1. A computer readable storage medium, comprising executable instructions to:

define a map-reduce document that coordinates processing of data in a distributed database, the map-reduce document complying with a map-reduce specification that integrates map-reduce functions with queries in a query language; and

execute operations specified by the map-reduce document in the distributed database.

2. The computer readable storage medium of claim 1, wherein the map-reduce document specifies an input source, a map function, a reduce function, and an output target.

3. The computer readable storage medium of claim 2, wherein the input source includes input sources selected from a file, a table in the distributed database, a query to the distributed database, and a operating system command.

4. The computer readable storage medium of claim 3, wherein an input source resides on a node that processes the input source.

5. The computer readable storage medium of claim 3, wherein an input source is distributed into segments across a plurality of nodes.

6. The computer readable storage medium of claim 5, wherein specifying the input source results in an automatic mapping to the segments across the plurality of nodes.

7. The computer readable storage medium of claim 2, wherein the map function is selected from a built-in map function and a user-defined map function.

8. The computer readable storage medium of claim 2, wherein the reduce function is selected from a built-in reduce function and a user-defined reduce function.

9. The computer readable storage medium of claim 7, wherein the map function comprises executable instructions to process a list of input parameters from the input source to generate a list of output parameters, the output parameters having a query language data type and a key.

10. The computer readable storage medium of claim 9, wherein the reduce function comprises a transition function with executable instructions to process the list of output parameters into an intermediate state output, the intermediate state output comprising accumulated values associated with the key.

11. The computer readable storage medium of claim 10 wherein the intermediate state output includes a transition output, a consolidate output and a finalize output.

12. The computer readable storage medium of claim 11, wherein the distributed database comprises a master database instance, a plurality of segment database instances, and a networking layer.

13. The computer readable storage medium of claim 12, further comprising executable instructions to produce the transition output at one or more of the plurality of segment database instances.

14. The computer readable storage medium of claim 12, further comprising executable instructions to provide consolidate output through the networking layer.

15. The computer readable storage medium of claim 3, further comprising executable instructions to define a task object for specifying an input source and a task object function.

16. The computer readable storage medium of claim 15, wherein the task object function comprises a map function.

17. The computer readable storage medium of claim 15, wherein the task object function comprises a reduce function.

18. The computer readable storage medium of claim 15, further comprising executable instructions to execute the task object function in a query to the distributed database.

19. The computer readable storage medium of claim 1, wherein the map-reduce document is processed by a core engine of the distributed database.

20. The computer readable storage medium of claim 1, wherein the map-reduce document is integrally processed during Structured Query Language (SQL) query processing.

21. The computer readable storage medium of claim 1 further comprising executable instructions to process data from a plurality of data sources selected from a file, a table in a database, a query and an operating system command.

22. The computer readable storage medium of claim 1 further comprising executable instructions to utilize queries to execute a map function and a reduce function.

23. The computer readable storage medium of claim 1 further comprising executable instructions to execute a map function and a reduce function as part of queries to the distributed database.