DISTRIBUTED STEAM PROCESSING

• US 20180063226A1
• Filed: 08/08/2017
• Published: 03/01/2018
• Est. Priority Date: 03/05/2009
• Status: Active Grant

• Current Assignee: PayPal, Inc. (PayPal Holdings, Inc.)
• Original Assignee: PayPal, Inc. (PayPal Holdings, Inc.)
• Inventors: Chiu, Chi-Hsien, Crane, Patrick, Neckorcuk, Alyssa, Singh, Gyanit, Sundaresan, Neelakantan
• Granted Patent: US 10,812,566 B2
• Time in Patent Office: Days
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method and system for forming hybrid cluster to process log files are described. In example embodiments, a method configures a node to execute as a first slave node. The first slave node executes in a first operating environment. The method also adds the first slave node to a Hadoop cluster. The Hadoop cluster includes a second slave node that operates in a second and different operating environment.

Claims

1. (canceled)

2. A system comprising:

one or more processors;

one or more non-transitory computer-readable media containing instructions that, in response to being executed by the one or more processors, cause the system to perform one or more operations comprising;

receive a request, from a first slave node, to access a file stored in a cluster of nodes, the request being based on a determination that the first slave node utilizes a first operating system, the cluster of nodes configured to utilize a second operating system different from the first operating system, the cluster of nodes including the first slave node that utilizes the first operating system and a plurality of second slave nodes that utilize the second operating system;

in response to receiving the request, determine that the file is not stored locally at the first slave node but is stored in association with a shared storage server accessible in the cluster of nodes via the second operating system;

in response to determining the file is not stored locally, request the file from the shared storage server via a server-side runner module, the server-side runner module configured to communicate with the shared storage server on behalf of the first slave node using the second operating system; and

in response to receiving the file from the server-side runner module, store the file locally in the first slave node such that the first slave node can directly access the file using the first operating system.

3. The system of claim 2, wherein the second operating system natively supports Hadoop.

4. The system of claim 3, wherein

the cluster of nodes includes a master node that utilizes the second operating system and is configured to distribute data among the first slave node and the plurality of second slave nodes;

and

the operations further comprise;

receive a server key from the master node, and

store the server key at the first slave node such that the first slave node accesses the cluster of nodes without a password.

5. The system of claim 2, wherein the operations further comprise:

send a message to add the first slave node to the cluster of nodes, the message being sent to the server-side runner module, and

update the shared storage server to indicate the first slave node as being added to the cluster of nodes.

6. The system of claim 2, wherein the operations further comprise:

send a message to remove the first slave node from the cluster of nodes, the message being sent to the server-side runner module, and

update the shared storage server to indicate the first slave node as being removed from the cluster of nodes.

7. The system of claim 2, wherein:

the first operating system executes an emulating program that emulates an environment of the second operating system; and

the first slave node is configured to make an operating system call that is natively supported by the second operating system based on the emulating program executed by the first operating system.

8. The system of claim 2, wherein the cluster of nodes processes log files of a content provider stored in the cluster of nodes, and the file is at least part of the log files.

9. A method comprising:

receiving a request, from a first slave node, to access a file stored in a cluster of nodes, the request being based on a determination that the first slave node utilizes a first operating system, the cluster of nodes configured to utilize a second operating system different from the first operating system, the cluster of nodes including the first slave node;

in response to receiving the request, determining that the file is not stored locally at the first slave node but is stored in association with a shared storage server accessible via the second operating system;

in response to determining the file is not stored locally, requesting the file from the shared storage server via a server-side runner module, the server-side runner module configured to communicate with the shared storage server using the second operating system and on behalf of the first slave node; and

in response to receiving the file from the server-side runner module, storing the file locally in the first slave node such that the first slave node can directly access the file using the first operating system.

10. The method of claim 9, wherein the second operating system natively supports Hadoop.

11. The method of claim 10, wherein:

the cluster of nodes includes a master node that utilizes the second operating system and is configured to distribute data among the first slave node and a plurality of second slave nodes, the plurality of second slave nodes utilizing the second operating system; and

the method further comprises;

receiving a server key from the master node, and

storing the server key at the first slave node such that the first slave node accesses the cluster of nodes without a password.

12. The method of claim 9, further comprising:

sending a message to add the first slave node to the cluster of nodes, the message being sent to the server-side runner module, and

updating the shared storage server to indicate the first slave node as being added to the cluster of nodes.

13. The method of claim 9, the method further comprising:

sending a message to remove the first slave node from the cluster of nodes, the message being sent to the server-side runner module, and

updating the shared storage server to indicate the first slave node as being removed from the cluster of nodes.

14. The method of claim 9, wherein the cluster of nodes processes log files of a content provider stored in the cluster of nodes, and the file is at least part of the log files.

15. A non-transitory computer-readable medium containing instructions that, in response to being executed by one or more processors, cause a machine to perform operations, the operations comprising:

receive a request, from a first slave node, to access a file stored in a cluster of nodes, the request being based on a determination that the first slave node utilizes a first operating system, the cluster of nodes configured to utilize a second operating system different from the first operating system, the cluster of nodes including the first slave node and a plurality of second slave nodes that utilize the second operating system;

in response to receiving the request, determine that the file is not stored locally at the first slave node but is stored in association with a shared storage server accessible within the cluster of nodes via the second operating system;

in response to determining the file is not stored locally, request the file from the shared storage server via a server-side runner module, the server-side runner module configured to communicate with the shared storage server using the second operating system; and

in response to receiving the file from the server-side runner module, store the file locally in the first slave node.

16. The computer-readable medium of claim 15, wherein the second operating system natively supports Hadoop.

17. The computer-readable medium of claim 16, wherein:

the cluster of nodes includes a master node that utilizes the second operating system, and is configured to distribute data among the first slave node and the plurality of second slave nodes; and

the operations further comprise;

receive a server key from the master node, and

store the server key at the first slave node such that the first slave node accesses the cluster of nodes without a password.

18. The computer-readable medium of claim 15, wherein the operations further comprise:

send a message to add the first slave node to the cluster of nodes, the message being sent to the server-side runner module, and

update the shared storage server to indicate the first slave node as being added to the cluster of nodes.

19. The computer-readable medium of claim 15, wherein the operations further comprise:

send a message to remove the first slave node from the cluster of nodes, the message being sent to the server-side runner module, and

update the shared storage server to indicate the first slave node as being removed from the cluster of nodes.

20. The computer-readable medium of claim 15, wherein:

the first operating system executes an emulating program that emulates an environment of the second operating system; and

the first slave node is configured to make an operating system call that is natively supported by the second operating system based on the emulating program executed by the first operating system.

21. The computer-readable medium of claim 15, wherein the cluster of nodes processes log files of a content provider stored in the cluster of nodes, and the file is at least part of the log files.