Integrating map-reduce into a distributed relational database

• US 8,713,038 B2
• Filed: 04/02/2009
• Issued: 04/29/2014
• Est. Priority Date: 04/02/2009
• Status: Active Grant

• Current Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Original Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Inventors: Cohen, Jeffrey Ira, Lonergan, Luke, Welton, Caleb E.
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Waldron, Scott A
• Application Number:
  US12/417,550
• Publication Number:
  US 20100257198A1
• Time in Patent Office: 1,853 Days
• Field of Search: 707/770, 707/792, 707/796, 707/764
• US Class Current: 707/764
• CPC Class Codes: G06F 16/24542 Plan optimisation
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 16/93 Document management systems

Abstract

A computer readable storage medium includes executable instructions to define a map-reduce document that coordinates processing of data in a distributed database. The map-reduce document complies with a map-reduce specification that integrates map-reduce functions with queries in a query language. The operations specified by the map-reduce document are executed in the distributed database.

Claims

1. A computer readable non-transitory storage medium for storing executable instructions, comprising executable instructions for controlling the operation of a computer in a distributed relational database to:

define a map-reduce document that coordinates processing of data from an input source in database segment hosts of the distributed relational database using map-reduce functions comprising map functions and reduce functions in a computer executable scripting language and database SQL queries in a structured query language (SQL), the map-reduce document specifying the map-reduce functions and the database SQL queries and integrating both the map-reduce functions and the database SQL queries in other structured query language queries for processing by said database segment hosts; and

submit said map-reduce document to a map-reduce program associated with a core engine of the distributed relational database that distributes said map-reduce functions to the database segment hosts distributed across said distributed relational database for execution as SQL queries of both the map-reduce functions and the SQL queries specified by the map-reduce document in said distributed relational database segment hosts.

2. The computer readable non-transitory storage medium of claim 1, wherein the map-reduce document specifies the input source, a map function, a reduce function, and an output target.

3. The computer readable non-transitory storage medium of claim 2, wherein the input source includes input sources selected from a file, a table in the distributed relational database, a SQL query to the distributed relational database, and an operating system command.

4. The computer readable non-transitory storage medium of claim 3, wherein the input source resides on a database node that processes the input source.

5. The computer readable non-transitory storage medium of claim 3, wherein the input source is distributed into database segments across a plurality of database nodes and is formed using a further SQL query to combine data from said database segments.

6. The computer readable non-transitory storage medium of claim 5, wherein specifying the input source results in an automatic mapping to the database segments across the plurality of nodes.

7. The computer readable non-transitory storage medium of claim 5 further comprising executable instructions to process data from a plurality of data sources selected from a file, a table in said database segments, a SQL query to the database segments, and an operating system command.

8. The computer readable non-transitory storage medium of claim 1, wherein the map function is selected from a built-in map function and a user-defined map function.

9. The computer readable non-transitory storage medium of claim 1, wherein the reduce function is selected from a built-in reduce function and a user-defined reduce function.

10. The computer readable non-transitory storage medium of claim 1, wherein the map-reduce functions comprise executable instructions to process a list of input parameters from the input source to generate a list of output parameters, the output parameters having a SQL query language data type and a key.

11. The computer readable non-transitory storage medium of claim 10, wherein the reduce function comprises a transition function with executable instructions to process the list of output parameters into an intermediate state output, the intermediate state output comprising accumulated state values associated with a key.

12. The computer readable non-transitory storage medium of claim 11 wherein the intermediate state output includes a transition output, a consolidate output that accumulates multiple state values into a single state value, and a finalize output.

13. The computer readable non-transitory storage medium of claim 1, wherein the distributed relational database comprises a master database instance, a plurality of segment database instances, and a networking layer.

14. The computer readable non-transitory storage medium of claim 13, further comprising executable instructions to produce a transition output at one or more of the plurality of segment database instances.

15. The computer readable non-transitory storage medium of claim 13, further comprising executable instructions to provide a consolidate output through the networking layer.

16. The computer readable non-transitory storage medium of claim 1, further comprising executable instructions to define a task object for specifying the input source and a task object function.

17. The computer readable non-transitory storage medium of claim 16, wherein the task object function comprises one of a map function and a reduce function.

18. The computer readable non-transitory storage medium of claim 16, further comprising including executable instructions to execute the task object function in a SQL query to the distributed relational database.

19. The computer readable non-transitory storage medium of claim 1, wherein the map-reduce document is integrally processed during Structured Query Language (SQL) query processing.

20. The computer readable non-transitory storage medium of claim 1 further comprising executable instructions to utilize the SQL queries to execute said map functions and said reduce functions.

21. The computer readable non-transitory storage medium of claim 1 further comprising executable instructions to execute said map functions and said reduce functions as part of the SQL queries to the distributed relational database.

22. The computer readable non-transitory storage media of claim 1, wherein said computer executable scripting language is selected from one of PERL and Python scripting languages.

23. A method of integrating map-reduce functions and queries in a distributed relational database, comprising:

defining a map-reduce document that coordinates processing of data from an input source in database segment hosts of the distributed relational database using map-reduce functions comprising map functions and reduce functions and database SQL queries in a structured query language (SQL), the map-reduce document specifying the map-reduce functions and the database SQL queries and integrating both the map-reduce functions and the database SQL queries in other structured query language queries for processing by said database segment hosts; and

submitting said map-reduce document to a map-reduce program associated with a core engine of the distributed relational database that distributes said map-reduce functions to the database segment hosts across said distributed relational database for execution as SQL queries of both the map-reduce functions and the SQL queries specified by the map-reduce document in said distributed relational database segment hosts.

24. The method of claim 23, wherein said defining said map-reduce document comprises defining a task object specifying the input source of data as one or more of a file, a table in the distributed relational database, a SQL query, and an operating system command, and specifying the map-reduce function as one or both of a map function and a reduce function that operate on the data from said input source.

25. The method of claim 23, wherein said distributed relational database is a parallel relational database, and said map-reduce functions and said SQL queries are user defined.