Methods and systems for parallel distributed computation

• US 9,396,023 B1
• Filed: 12/30/2014
• Issued: 07/19/2016
• Est. Priority Date: 12/30/2014
• Status: Active Grant

• Current Assignee: Marvell Asia Pte Limited (Marvell Technology Group Limited)
• Original Assignee: QLogic Corporation (Marvell Technology Group Limited)
• Inventors: Trivedi, Hemant
• Primary Examiner(s): Chavis, John
• Application Number:
  US14/585,761
• Time in Patent Office: 567 Days
• Field of Search: 718/106
• US Class Current: 1/1
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: G06F 9/54 Interprogram communication

Abstract

Methods and systems for performing a shared computing task are provided. The method includes configuring a shared computing task among a plurality of computing nodes each executing an application for performing the shared computing task; generating intermediate data by each of the plurality of computing nodes during a mapping process for performing the shared computing task; capturing the intermediate data for each of the plurality of computing nodes; storing the intermediate data for each of the plurality of computing nodes at a key-value data store at a shared storage device that enables searching for the intermediate data using a key-value pair; and using the key-value pair to obtain the intermediate data when needed for a reduce process of the shared computing task.

Claims

1. A machine implemented method, comprising:

configuring a shared computing task among a plurality of computing nodes each computing node executing an application for performing the shared computing task;

generating intermediate data by each of the plurality of computing nodes during a mapping process for performing the shared computing task;

capturing the intermediate data for each of the plurality of computing nodes and transferring the intermediate data of each of the plurality of computing nodes to a shared storage device;

storing the intermediate data for each of the plurality of computing nodes at a key-value data store at the shared storage device that enables searching for the intermediate data of any node at the shared storage device using a key-value pair; and

using the key-value pair to obtain the intermediate data from the shared storage device when needed for a reduce process of the shared computing task;

wherein the mapping process and the reduce process are executed by a processor coupled to a memory device.

2. The method of claim 1, wherein a filter driver executed by each of the computing node interfaces with the mapping process to capture the intermediate data generated by its node and sends the intermediate data to a message module for sending it to the shared storage device.

3. The method of claim 2, wherein a shuffle plugin module executed by the processor obtains the intermediate data from the shared storage using the key-value pair and provides the intermediate data to the reduce process, when needed.

4. The method of claim 1, wherein the plurality of computing nodes operate within a Hadoop architecture for executing the shared computing task.

5. The method of claim 1, wherein each computing node interfaces with a converged network adapter that processes both network and storage traffic to transfer the intermediate data to the shared storage.

6. The method of claim 1, wherein the shared storage is managed by an intelligent storage adapter that can access the shared storage and other storage via a storage area network.

7. The method of claim 6, wherein the intelligent storage adapter has access to a solid state storage device for storing the intermediate data.

8. A non-transitory, machine readable storage medium having stored thereon instructions for performing a method, comprising machine executable code which when executed by at least one machine, causes the machine to:

configure a shared computing task among a plurality of computing nodes each computing node executes an application for performing the shared computing task;

generate intermediate data by each of the plurality of computing nodes during a mapping process for performing the shared computing task;

capture the intermediate data for each of the plurality of computing nodes and transfer the intermediate data of each of the plurality of computing nodes to a shared storage device;

store the intermediate data for each of the plurality of computing nodes at a key-value data store at the shared storage device that enables searching for the intermediate data of any node at the shared storage device using a key-value pair; and

use the key-value pair to obtain the intermediate data from the shared storage device when needed for a reduce process of the shared computing task;

wherein the mapping process and the reduce process are executed by a processor coupled to a memory device.

9. The non-transitory, storage medium of claim 8, wherein a filter driver executed by each of the computing node interfaces with the mapping process to capture the intermediate data and sends the intermediate data to a message module for sending it to the shared storage device.

10. The non-transitory, storage medium of claim 9, wherein a shuffle plugin module executed by the processor obtains the intermediate data from the shared storage using the key-value pair and provides the intermediate data to the reduce process, when needed.

11. The non-transitory, storage medium of claim 8, wherein the plurality of computing nodes operate within a Hadoop architecture for executing the shared computing task.

12. The non-transitory, storage medium of claim 8, wherein each computing node interfaces with a converged network adapter that processes both network and storage traffic to transfer the intermediate data to the shared storage.

13. The non-transitory, storage medium of claim 8, wherein the shared storage is managed by an intelligent storage adapter that can access the shared storage and other storage via a storage area network.

14. The non-transitory, storage medium of claim 13, wherein the intelligent storage adapter has access to a solid state storage device for storing the intermediate data.

15. A system, comprising:

a memory containing machine readable medium comprising machine executable code having stored thereon instructions; and

a processor module coupled to the memory, the processor module configured to execute the machine executable code to;

configure a shared computing task among a plurality of computing nodes each computing node executes an application for performing the shared computing task;

generate intermediate data by each of the plurality of computing nodes during a mapping process for performing the shared computing task;

capture the intermediate data for each of the plurality of computing nodes and transfer the intermediate data of each of the plurality of computing nodes to a shared storage device;

store the intermediate data for each of the plurality of computing nodes at a key-value data store at the shared storage device that enables searching for the intermediate data of any node at the shared storage device using a key-value pair; and

use the key-value pair to obtain the intermediate data from the shared storage device when needed for a reduce process of the shared computing task;

wherein the mapping process and the reduce process are executed by a processor coupled to a memory device.

16. The system of claim 15, wherein a filter driver executed by each of the computing node interfaces with the mapping process to capture the intermediate data and sends the intermediate data to a message module for sending it to the shared storage device.

17. The system of claim 16, wherein a shuffle plugin module executed by the processor obtains the intermediate data from the shared storage using the key-value pair and provides the intermediate data to the reduce process, when needed.

18. The system of claim 15, wherein the plurality of computing nodes operate within a Hadoop architecture for executing the shared computing task.

19. The system of claim 15, wherein each computing node interfaces with a converged network adapter that processes both network and storage traffic to transfer the intermediate data to the shared storage.

20. The system of claim 15, wherein the shared storage is managed by an intelligent storage adapter that can access the shared storage and other storage via a storage area network.