Distributed storage of data

• US 9,953,071 B2
• Filed: 09/09/2014
• Issued: 04/24/2018
• Est. Priority Date: 09/10/2013
• Status: Active Grant

• Current Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Original Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Inventors: Vasu, Arun, Varghese, Abraham, Sreekumar, Akhil
• Primary Examiner(s): LEROUX, ETIENNE PIERRE
• Application Number:
  US14/481,440
• Publication Number:
  US 20150074115A1
• Time in Patent Office: 1,323 Days
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A file generation system for storage of structured data onto a distributed database includes an intermediate data generation module to generate a set of intermediate key-value pairs for each of one or more records in at least one subset of the structured data. A key-value pair includes a key and a value corresponding to the key, where the key is a unique identifier of the value. A file generation system further includes an intermediate data sorting module to sort the set of intermediate key-value pairs to generate a plurality of output files. Each of the plurality of output files includes at least one key-value pair. Further, the file generation system includes a file storing module to store the plurality of output files in the distributed database, where the plurality of output files are representative of the structured data.

Claims

1. A computer-implemented file generation system for storage of structured data onto a distributed database, the file generation system comprising:

a processor;

an intermediate data generation module coupled to the processor to generate a set of intermediate key-value pairs for each of one or more records in at least one subset of the structured data, wherein a key-value pair includes a key and a value corresponding to the key, and wherein the key is a unique identifier of the value;

an intermediate data sorting module coupled to the processor to sort the set of intermediate key-value pairs to generate a plurality of output files, wherein each of the plurality of output files includes at least one key-value pair, wherein generating the intermediate key-value pairs and sorting the intermediate key-value pairs is performed by one node without transferring the intermediate key-value pairs over a network;

a data segregation module coupled to the processor to obtain, from a user device, the structured data to be stored in a distributed database over the network, wherein the structured data is organized as logical records and is identified by a unique key, and wherein each logical record is assigned to a mapper; and

a file storing module coupled to the processor to store the plurality of output files in the distributed database, wherein the plurality of output files is representative of the structured data.

2. The computer-implemented file generation system as claimed in claim 1, wherein the data segregation module segregates the structured data into the at least one subset, wherein the at least one subset includes one or more records.

3. The computer-implemented file generation system as claimed in claim 1, wherein the intermediate data sorting module sorts the set of intermediate key-value pairs based on the key.

4. The computer-implemented file generation system as claimed in claim 1, wherein the intermediate data sorting module sorts the set of intermediate key-value pairs lexicographically.

5. The computer-implemented file generation system as claimed in claim 1, wherein the file storing module further stores each of the plurality of output files as at least one HFile.

6. The computer-implemented file generation system as claimed in claim 5, wherein the at least one HFile includes one or more key-value pairs, wherein keys corresponding to the one or more key-value pairs are same.

7. The computer-implemented file generation system as claimed in claim 1, wherein the file storing module stores the plurality of output files as at least one HFile into an HBase.

8. The computer-implemented file generation system as claimed in claim 1, the file storing module further invokes a bulk load application programming interface (API) to store the plurality of output files in the distributed database.

9. The computer-implemented file generation system as claimed in claim 1, wherein the distributed database is an HBase instance in a Hadoop cluster.

10. A computer-implemented method for storage of structured data onto a distributed database, the method comprising:

obtaining, from a user device, a plurality of subsets of the structured data to be stored in the distributed database, wherein each of the plurality of subsets includes one or more records;

generating a set of intermediate key-value pairs for each of the one or more records in each of the plurality of subsets of the structured data, wherein a key-value pair includes a key and a value corresponding to the key, and wherein the key is a unique identifier of the value;

sorting the set of intermediate key-value pairs to generate a plurality of output files, wherein each of the plurality of output files includes at least one key-value pair, wherein generating the intermediate key-value pairs and sorting the intermediate key-value pairs is performed by one node without transferring the intermediate key-value pairs over a network;

obtaining, from a user device, the structured data to be stored in a distributed database over the network, wherein the structured data is organized as logical records and is identified by a unique key, and wherein each logical record is assigned to a mapper; and

storing the plurality of output files in the distributed database, wherein the plurality of output files is representative of the structured data.

11. The computer-implemented method as claimed in claim 10, wherein the method further comprises segregating the structured data to generate the plurality of subsets.

12. The computer-implemented method as claimed in claim 10, wherein the distributed database is an HBase instance in a Hadoop cluster.

13. The computer-implemented method as claimed in claim 12, wherein each of the plurality of output files is further stored as at least one HFile into the Hadoop cluster, wherein the at least one HFile includes one or more key-value pairs, and wherein keys corresponding to the one or more key-value pairs are same.

14. The computer-implemented method as claimed in claim 10, wherein method further comprises storing the plurality of output files into an HBase.

15. A non-transitory computer-readable medium having embodied thereon a computer program for executing a method comprising:

obtaining, from a user device, a plurality of subsets of the structured data to be stored in the distributed database, wherein each of the plurality of subsets includes one or more records;

generating a set of intermediate key-value pairs for each of the one or more records in each of the plurality of subsets of the structured data, wherein a key-value pair includes a key and a value corresponding to the key, and wherein the key is a unique identifier of the value;

sorting the set of intermediate key-value pairs to generate a plurality of output files, wherein each of the plurality of output files includes at least one key-value pair, wherein generating the intermediate key-value pairs and sorting the intermediate key-value pairs is performed by one node without transferring the intermediate key-value pairs over a network;

obtaining, from a user device, the structured data to be stored in a distributed database over the network, wherein the structured data is organized as logical records and is identified by a unique key, and wherein each logical record is assigned to a mapper;

storing the plurality of output files in the distributed database, wherein the plurality of output files is representative of the structured data.