Apparatus and Method for Securing Preliminary Information About Database Fragments for Utilization in Mapreduce Processing

• US 20140114994A1
• Filed: 10/24/2012
• Published: 04/24/2014
• Est. Priority Date: 10/24/2012
• Status: Active Grant

• Current Assignee: MarkLogic Corporation (Progress Software Corporation)
• Original Assignee: MarkLogic Corporation (Progress Software Corporation)
• Inventors: Lindblad, Christopher, Chen, Jane X.
• Granted Patent: US 8,892,599 B2
• Time in Patent Office: Days
• US Class Current: 707/756
• CPC Class Codes: G06F 16/2471 Distributed queries

Abstract

A method of processing a query in a distributed database implemented across a set of nodes includes receiving a query. The query is divided into split characterization queries. The split characterization queries are distributed to worker nodes. Each worker node stores a partition of the distributed database with encoded textual objects and pre-defined indices characterizing encoded textual object fragments. The split characterization queries are executed at the worker nodes to obtain preliminary information about query results. Executing the split characterization queries includes matching query fragments associated with the split characterization queries with encoded textual object fragments of the pre-defined indices to produce fragment matches representative of the size of the query results. For each split characterization query the preliminary information about query results includes a fragment count, a database partition identification, and a database host name.

Claims

1. A method of processing a query in a distributed database implemented across a plurality of nodes, comprising:

receiving a query;

dividing the query into split characterization queries;

distributing the split characterization queries to a plurality of worker nodes, wherein each worker node stores a partition of the distributed database as a database with encoded textual objects and pre-defined indices characterizing encoded textual object fragments; and

executing the split characterization queries at the plurality of worker nodes to obtain preliminary information about query results, wherein executing the split characterization queries includes matching query fragments associated with the split characterization queries with encoded textual object fragments to produce fragment matches representative of the size of the query results, wherein for each split characterization query the preliminary information about query results includes a fragment count, a database partition identification, and a database host name.

2. The method of claim 1 wherein the preliminary information further includes a fragment offset.

3. The method of claim 1 wherein the preliminary information further includes a fragment position range utilized in connection with the pre-defined indices during execution of a map task.

4. The method of claim 1 further comprising associating an individual input split with a database partition.

5. The method of claim 4 wherein each database partition has an associated worker node host.

6. The method of claim 4 wherein the input splithas subsets of input data assigned to a map task.

7. The method of claim 6 further comprising executing the map task, wherein the map task transforms data in an input split into a set of output key-value pairs.

8. The method of claim 7 further comprising processing the output key-value pairs as a reduce task, wherein the reduce task produces a set of final result key-value pairs.

9. A non-transitory computer readable storage medium, comprising executable instructions to:

execute a split characterization query against a partition of a distributed database with encoded textual objects and pre-defined indices characterizing encoded textual object fragments, wherein the executable instructions to execute include executable instructions to match query fragments associated with the split characterization query with encoded textual object fragments of the pre-defined indices to produce fragment matches representative of the size of query results; and

generate preliminary information about the query results, wherein the preliminary information includes a fragment count, a database partition identification, and a database host name.

10. The non-transitory computer readable storage medium of claim 9 further comprising executable instructions to execute a map task configured based upon the preliminary information.

11. The non-transitory computer readable storage medium of claim 10 wherein the map task associates an input split with a database partition.

12. The non-transitory computer readable storage medium of claim 11 wherein the database partition has an associated worker node host.

13. The non-transitory computer readable storage medium of claim 10 further comprising executable instructions to execute a reduce task.

14. The non-transitory computer readable storage medium of claim 9 further comprising an application program interface to receive the split characterization query.

15. The non-transitory computer readable storage medium of claim 9 further comprising an application program interface to convey the preliminary information to a MapReduce map task module.

16. The non-transitory computer readable storage medium of claim 9 wherein the preliminary information further includes a fragment position range utilized in connection with the pre-defined indices during execution of a map task.