Systems and methods for massive structured data management over cloud aware distributed file system

• US 8,775,425 B2
• Filed: 08/24/2010
• Issued: 07/08/2014
• Est. Priority Date: 08/24/2010
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gupta, Himanshu, Gupta, Rajeev, Mohania, Mukesh Kumar, Nambiar, Ullas Balan
• Primary Examiner(s): Corrielus, Jean M
• Application Number:
  US12/862,273
• Publication Number:
  US 20120054182A1
• Time in Patent Office: 1,414 Days
• Field of Search: 707/737, 707/770, 707/802, 707/714, 707/602, 707/713, 707/792, 707/719, 707/757
• US Class Current: 707/737
• CPC Class Codes: G06F 16/24532 of parallel queries

Abstract

Methods and arrangements for accommodating a query, directing the query to datasets, creating partitions and partitioning the datasets, and returning a response to the query, the response being structured in accordance with the created partitions.

Claims

1. A method comprising:

accommodating a query;

directing the query to datasets which include data from files in a distributed file system;

creating partitions of the datasets, wherein said creating comprises creating smart replicas, and wherein the smart replicas comprise reordered content of the datasets;

indexing the partitions via mapping partition keys to a database atop the distributed file system;

parsing the query;

looking up indexed values based on the parsed query;

identifying partitions based on said looking up of indexed values;

rewriting the query, with partition information, into the database atop the distributed file system; and

returning a response to the query, wherein the response is structured based on the created partitions.

2. The method according to claim 1, wherein said creating of partitions comprises creating horizontal partitions.

3. The method according to claim 2, wherein the horizontal partitions comprise tuples of high similarity.

4. The method according to claim 1, wherein the datasets include data from a data warehouse.

5. The method according to claim 1, comprising repartitioning each partition locally.

6. The method according to claim 1, wherein said indexing comprises employing inverted index maps identifying pairs where attributes are equivalent to predetermined values.

7. The method according to claim 6, wherein said employing of inverted index maps comprises indicating occurrences of attribute-value equivalencies among partitions.

8. The method according to claim 1, wherein said creating of smart replicas comprises keeping replicas sorted over different attribute sets.

9. An apparatus comprising:

one or more processors; and

a computer readable storage medium having computer readable program code embodied therewith and executable by the one or more processors, the computer readable program code being configured to;

accommodate a query;

direct the query to datasets which include data from files in a distributed file system;

create partitions of the datasets, wherein to create partitions comprises creating smart replicas, and wherein the smart replicas comprise reordered content of the datasets;

index the partitions via mapping partition keys to a database atop the distributed file system;

parse the query;

look up indexed values based on the parsed query;

identify partitions based on the looking up of indexed values;

rewrite the query, with partition information, into the database atop distributed file system; and

return a response to the query, wherein the response is structured based on the created partitions.

10. A computer program product embedded in a computer readable storage medium embodied with computer readable program code which, when executed, causes a computing device to perform operations, the computer readable program code being configured to:

accommodate a query;

direct the query to datasets which include data from files in a distributed file system;

create partitions of the datasets, wherein to create partitions comprises creating smart replicas, and wherein the smart replicas comprise reordered content of the datasets;

index the partitions via mapping partition keys to a database atop the distributed file system;

parse the query;

look up indexed values based on the parsed query;

identify partitions based on the looking up of indexed values;

rewrite the query, with partition information, into the database atop distributed file system; and

return a response to the query, wherein the response is structured based on the created partitions.

11. The computer program product according to claim 10, wherein said computer readable program code is configured to create horizontal partitions.

12. The computer program product according to claim 10, wherein the datasets include data from a data warehouse.

13. The computer program product according to claim 10, wherein said computer readable program code is configured to re-partition each partition locally.

14. The computer program product according to claim 10, wherein said computer readable program code is configured to employ inverted index maps identifying pairs where attributes are equivalent to predetermined values.

15. The computer program product according to claim 14, wherein said computer readable program code is configured to indicate occurrences of attribute-value equivalencies among partitions.

16. The computer program product according to claim 10, wherein said computer readable program code is configured to keep replicas sorted over different attribute sets.