SMARTER BIG DATA PROCESSING USING COLLABORATIVE MAP REDUCE FRAMEWORKS

• US 20140379632A1
• Filed: 06/19/2013
• Published: 12/25/2014
• Est. Priority Date: 06/19/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kulkarni, Shrinivas S., Gera, Bharat K.
• Granted Patent: US 9,448,851 B2
• Time in Patent Office: Days
• US Class Current: 707/608
• CPC Class Codes: G06F 9/5061 Partitioning or combining of resources
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services

Abstract

Map reduce processing is performed holistically, to yield map reduce results data, on an aggregated group of data sets, which had been generated and maintained with “splits” separating the separate date sets, rather than being performed separately on the split data sets and then aggregating (or attempting to aggregate) the map reduce processing results.

Claims

1. A method comprising:

receiving a first collaborative data set;

receiving a first non-collaborative data set;

combining at least the first collaborative data set and the first non-collaborative data set to make a combined map reduce (MR) data set; and

performing MR processing on the combined MR data set to make MR results data.

2. The method of claim 1 wherein at least the combining and the performing MR processing are performed by computer software running on computer hardware.

3. The method of claim 1 further comprising at least one of the following:

outputting the MR results data; and

saving the MR results data to a storage device.

4. The method of claim 1 further comprising:

receiving a second collaborative data set; and

combining at least the first collaborative data set, the first non-collaborative data set and the second non-collaborative data set to make a combined map reduce (MR) data set.

5. The method of claim 1 wherein the first collaborative data set is collected from one of the following sources:

a social media website, and a company intranet.

6. A computer program product comprising software stored on a software storage device, the software comprising:

first program instructions programmed to receive a first collaborative data set;

second program instructions programmed to receive a first non-collaborative data set;

third program instructions programmed to combine at least the first collaborative data set and the first non-collaborative data set to make a combined map reduce (MR) data set; and

fourth program instructions programmed to perform MR processing on the combined MR data set to make MR results data.

7. The product of claim 6 wherein the software is stored on a software storage device in a manner less transitory than a signal in transit.

8. The product of claim 6 wherein the software further comprises at least one of the following:

fifth program instructions programmed to output the MR results data; and

sixth program instructions programmed to save the MR results data to a storage device.

9. The product of claim 6 wherein the software further comprises:

fifth program instructions programmed to receive a second collaborative data set; and

sixth program instructions programmed to combine at least the first collaborative data set, the first non-collaborative data set and the second non-collaborative data set to make a combined map reduce (MR) data set.

10. The product of claim 6 wherein the first collaborative data set is collected from one of the following sources:

a social media website, and a company intranet.

11. A computer system comprising:

a processor(s) set; and

a software storage device;

wherein;

the processor set is structured, located, connected and/or programmed to run software stored on the software storage device; and

the software comprises;

first program instructions programmed to receive a first collaborative data set;

second program instructions programmed to receive a first non-collaborative data set;

third program instructions programmed to combine at least the first collaborative data set and the first non-collaborative data set to make a combined map reduce (MR) data set; and

fourth program instructions programmed to perform MR processing on the combined MR data set to make MR results data.

12. The system of claim 11 wherein the software further comprises at least one of the following:

fifth program instructions programmed to output the MR results data; and

sixth program instructions programmed to save the MR results data to a storage device.

13. The system of claim 11 wherein the software further comprises:

fifth program instructions programmed to receive a second collaborative data set; and

sixth program instructions programmed to combine at least the first collaborative data set, the first non-collaborative data set and the second non-collaborative data set to make a combined map reduce (MR) data set.

14. The system of claim 11 wherein the first collaborative data set is collected from one of the following sources:

a social media website, and a company intranet.