DISTRIBUTED COMPUTING SYSTEM FOR LARGE-SCALE DATA HANDLING

• US 20100162230A1
• Filed: 12/24/2008
• Published: 06/24/2010
• Est. Priority Date: 12/24/2008
• Status: Abandoned Application

• Current Assignee: Oath Inc. (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Chen, Peiji, Lin, Long Ji, Sordo, Mark, Zhang, Danny, Swanson, Donald
• Application Number:
  US12/343,979
• Publication Number:
  US 20100162230A1
• Time in Patent Office: Days
• US Class Current: 717/177
• CPC Class Codes: G06F 9/5072 Grid computing

Abstract

A method for processing data on a distributed computing environment is provided. Input data that is to be processed may be stored on an input storage module. Mapper code can be loaded onto a map module and executed. The mapper code can load a mapper executable file onto the map module from a central storage unit and instantiate the mapper executable file. The mapper code, then, can pass the input data to the mapper executable file. The mapper executable file can generate mapped data based on the input data and pass the mapped data back to the mapper code.

Claims

1. a system for processing data on a distributed computing environment, the system comprising:

a input data storage module containing input data from a weblog;

a map module in communication with the input data storage module to receive a split of the input data and configured to execute mapper code for manipulating the input data to generate mapped data.

a reduce module in communication with the map module to receive the map module to receive the mapped data, the reduce module being configured to execute reducer code for analyzing the mapped data and generate result data.

a result data storage module in communication with the reduce module to receive the result data from the reduce module.

a master module for coordinating the selection, set-up, and data flow of the map module and the reduce module, the master module loading the mapper code onto the mapper module and the reducer code onto the reducer module; and

a central storage module containing a mapper executable file and a reducer executable file, wherein the mapper code accesses the central storage module and loads the mapper executable file onto the mapper module and the reducer code loads the reducer executable file onto the reducer module.

2. The system for according to claim 1, wherein the mapper code instantiates the mapper executable file and the mapper executable file initiate a stream for communicating between the mapper code and the mapper executable file.

3. The system for according to claim 2, wherein the mapper code passes the input data to the mapper executable file through the stream in key/value format and the mapper executable file pass the mapped data to the mapper code through the stream in key/value format.

4. The system for according to claim 1, wherein the input data is impression\IP address data.

5. The system for according to claim 4, wherein the mapped data is impression\geographic region data.

6. The system for according to claim 5, wherein the result data is statistical data regarding a geographical region.

7. A method for processing data on a distributed computing environment, the method comprising:

storing input data from a weblog on an input storage module;

loading mapper code onto a map module through a master module;

executing the mapper code on the map module;

loading a mapper executable file onto the map module from a central storage module;

instantiating the mapper executable file on the map module;

retrieving a split of the input data from the input storage module;

passing the input data from the mapper code to the mapper executable file;

manipulating the input data to generate mapped data;

passing the mapped data from the mapper executable file to the mapper code;

loading reducer code onto a reduce module through a master module;

executing the reducer code on the reduce module;

loading a reducer executable file onto the reduce module from a central storage module;

instantiating the reducer executable file on the reduce module;

receiving the mapped data from the map module;

passing the input data from the reducer code to the reducer executable file;

manipulating the mapped data to generate result data;

passing the result data from the reducer executable file to the reducer code; and

storing the result data from the reducer on a result storage module.

8. The method for according to claim 7, wherein the input data is impression\IP address data.

9. The method for according to claim 8, wherein the mapped data is impression\geographic region data.

10. The method for according to claim 9, wherein the result data is statistical data regarding a geographical region.

11. A method for processing data on a distributed computing environment, the method comprising:

storing input data on an input storage module;

loading mapper code onto a map module;

executing the mapper code on the map module;

loading a mapper executable file onto the map module from a central storage module through the mapper code;

instantiating the mapper executable file on the map module;

retrieving a split of the input data from the input storage module;

passing the input data from the mapper code to the mapper executable file;

manipulating the input data to generate mapped data; and

passing the mapped data from the mapper executable file to the mapper code.

12. The method for according to claim 11, wherein the mapper code is a unix shell script.

13. The method for according to claim 11, further comprising loading a mapper library file onto the map module from the central storage module.

14. The method for according to claim 11, further comprising loading a mapper data file onto the map module from the central storage module.

15. The method for according to claim 14, wherein the mapper executable file generates the mapped data from the input data based on the mapper data file.

16. The method for according to claim 15, wherein the mapper data file is a look up table.

17. The method for according to claim 11, wherein the mapper executable file creates a data stream when instantiated and passes a pointer to the stream back to the mapper code.

18. The method for according to claim 14, wherein the input data is passed to the mapper executable over the stream in key/value format and the mapped data is passed to the mapper code over the stream in key/value format.

19. The method for according to claim 11, further comprising:

loading reducer code onto a reduce module;

executing the reducer code on the reduce module;

loading a reducer executable file onto the reduce module from a central storage module through the reducer code;

instantiating the reducer executable file on the reduce module;

receiving the mapped data from the map module;

passing the input data from the reducer code to the reducer executable file;

manipulating the mapped data to generate result data;

passing the result data from the reducer executable file to the reducer code; and

storing the result data from the reducer on a result storage module.

20. A computer readable medium having stored therein instructions executable by a programmed processor for ranking results, the computer readable medium comprising instructions for:

storing input data from a weblog on an input storage module;

loading mapper code onto a map module;

executing the mapper code on the map module;

loading a mapper executable file onto the map module from a central storage module using a fetch instruction in the mapper code;

instantiating the mapper executable file on the map module;

retrieving a split of the input data from the input storage module;

passing the input data from the mapper code to the mapper executable file;

manipulating the input data to generate mapped data;

passing the mapped data from the mapper executable file to the mapper code;

loading reducer code onto a reduce module;

executing the reducer code on the reduce module;

loading a reducer executable file onto the reduce module from a central storage module using a fetch instruction in the reducer code;

instantiating the reducer executable file on the reduce module;

receiving the mapped data from the map module;

passing the input data from the reducer code to the reducer executable file;

manipulating the mapped data to generate result data;

passing the result data from the reducer executable file to the reducer code; and

storing the result data from the reducer on a result storage module.

21. The method for according to claim 20, wherein the input data is impression\IP address data.

22. The method for according to claim 22, wherein the mapped data is impression\geographic region data.

23. The method for according to claim 23, wherein the result data is statistical data regarding a geographical region.