DISTRIBUTED STORAGE OF DATA

• US 20150074115A1
• Filed: 09/09/2014
• Published: 03/12/2015
• Est. Priority Date: 09/10/2013
• Status: Active Grant

• Current Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Original Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Inventors: Vasu, Arun, Varghese, Abraham, Sreekumar, Akhil
• Granted Patent: US 9,953,071 B2
• Time in Patent Office: Days
• US Class Current: 707/741
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A file generation system for storage of structured data onto a distributed database includes an intermediate data generation module to generate a set of intermediate key-value pairs for each of one or more records in at least one subset of the structured data. A key-value pair includes a key and a value corresponding to the key, where the key is a unique identifier of the value. A file generation system further includes an intermediate data sorting module to sort the set of intermediate key-value pairs to generate a plurality of output files. Each of the plurality of output files includes at least one key-value pair. Further, the file generation system includes a file storing module to store the plurality of output files in the distributed database, where the plurality of output files are representative of the structured data.

Claims

1. A file generation system for storage of structured data onto a distributed database, the file generation system comprising:

a processor;

an intermediate data generation module coupled to the processor to generate a set of intermediate key-value pairs for each of one or more records in at least one subset of the structured data, wherein a key-value pair includes a key and a value corresponding to the key, and wherein the key is a unique identifier of the value;

an intermediate data sorting module coupled to the processor to sort the set of intermediate key-value pairs to generate a plurality of output files, wherein each of the plurality of output files includes at least one key-value pair; and

a file storing module coupled to the processor to store the plurality of output files in the distributed database, wherein the plurality of output files is representative of the structured data.

2. The file generation system as claimed in claim 1 further comprises a data segregation module to obtain, from a user device, the structured data to be stored in the distributed database.

3. The file generation system as claimed in claim 2, wherein the data segregation module segregates the structured data into the at least one subset, wherein the at least one subset includes one or more records.

4. The file generation system as claimed in claim 1, wherein the intermediate data sorting module sorts the set of intermediate key-value pairs based on the key.

5. The file generation system as claimed in claim 1, wherein the intermediate data sorting module sorts the set of intermediate key-value pairs lexicographically.

6. The file generation system as claimed in claim 1, wherein the file storing module further stores each of the plurality of output files as at least one HFile.

7. The file generation system as claimed in claim 6, wherein the at least one HFile includes one or more key-value pairs, wherein keys corresponding to the one or more key-value pairs are same.

8. The file generation system as claimed in claim 1, wherein the file storing module stores the plurality of output files as at least one HFile into an HBase.

9. The file generation system as claimed in claim 1, the file storing module further invokes a bulk load application programming interface (API) to store the plurality of output files in the distributed database.

10. The file generation system as claimed in claim 1, wherein the distributed database is an HBase instance in a Hadoop cluster.

11. A method for storage of structured data onto a distributed database, the method comprising:

obtaining, from a user device, a plurality of subsets of the structured data to be stored in the distributed database, wherein each of the plurality of subsets includes one or more records;

generating a set of intermediate key-value pairs for each of the one or more records in each of the plurality of subsets of the structured data, wherein a key-value pair includes a key and a value corresponding to the key, and wherein the key is a unique identifier of the value;

sorting the set of intermediate key-value pairs to generate a plurality of output files, wherein each of the plurality of output files includes at least one key-value pair; and

storing the plurality of output files in the distributed database, wherein the plurality of output files is representative of the structured data.

12. The method as claimed in claim 11, wherein the method further comprises segregating the structured data to generate the plurality of subsets.

13. The method as claimed in claim 11, wherein the distributed database is an HBase instance in a Hadoop cluster.

14. The method as claimed in claim 13, wherein each of the plurality of output files is further stored as at least one HFile into the Hadoop cluster, wherein the at least one HFile includes one or more key-value pairs, and wherein keys corresponding to the one or more key-value pairs are same.

15. The method as claimed in claim 11, wherein method further comprises storing the plurality of output files into an HBase.

16. A non-transitory computer-readable medium having embodied thereon a computer program for executing a method comprising:

obtaining, from a user device, a plurality of subsets of the structured data to be stored in the distributed database, wherein each of the plurality of subsets includes one or more records;

generating a set of intermediate key-value pairs for each of the one or more records in each of the plurality of subsets of the structured data, wherein a key-value pair includes a key and a value corresponding to the key, and wherein the key is a unique identifier of the value;

sorting the set of intermediate key-value pairs to generate a plurality of output files, wherein each of the plurality of output files includes at least one key-value pair; and

storing the plurality of output files in the distributed database, wherein the plurality of output files is representative of the structured data.