Table data persistence

• US 9,639,544 B1
• Filed: 10/28/2014
• Issued: 05/02/2017
• Est. Priority Date: 10/28/2013
• Status: Active Grant

• Current Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Original Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Inventors: Smith, Daniel Allen, Baker, Anthony M., Bhanawat, Hemant, Ramnarayanan, Jagannathan, Bawaskar, Swapnil Prakash, Agrawal, Ashvin, Wale, Sumedh, Kumar, Neeraj
• Primary Examiner(s): ALAM, SHAHID AL
• Application Number:
  US14/526,347
• Time in Patent Office: 917 Days
• Field of Search: 707827
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a distributed parallel processing database system that persists table data in memory to a distributed file system. A distributed parallel processing database system persists table data in memory to a distributed file system. A parameter of creating a database table specifies that data records in the database table and history of changes to the data records can be stored in memory as well as in the distributed file system. When the database table is populated or otherwise modified, data records and the history in memory are evicted to the distributed file system as log files and removed from memory. The log files can be designated as write-only, where the data records, once written, cannot be read by structured query language (SQL) queries, or as read-write, where the data records, once written, can be read by SQL queries.

Claims

1. A method, comprising:

receiving, by a server cluster, a specification of a database table, the specification including an identifier of a data store on a distributed file system, wherein the server cluster comprises a plurality of servers and a plurality of computers, each server being a computer program executing on one or more of the computers, the server cluster being different from the distributed file system;

creating, by the servers in the server cluster, the database table based on the specification, the database table being distributed among each server of the server cluster;

storing, by each server in the server cluster, a history of modification of a portion of the database table distributed to that server as table data in a queue of that server, the history including events of inserting data records into the database table and events of changing data records in the database table, wherein the queue of that server and data records of the database table that are distributed to that server are stored on a storage device of the one or more computers allocated to that server;

determining, by at least one server of the server cluster, that a table data persistence condition has been satisfied on the at least one server;

in response, removing, by the at least one server, at least a portion of the queue from a storage device associated with the at least one server or removing at least a portion of the data records from the storage device associated with the at least one server; and

storing the removed portion of the queue or the removed portion of data records as log files in the data store of the distributed file system.

2. The method of claim 1, wherein the specification includes a parameter specifying that the table data be stored in the data store in a write-only mode, wherein the data stored in the write-only mode is ineligible for direct access by a structured query language (SQL) query statement issued to the server cluster.

3. The method of claim 1, wherein the specification includes a parameter specifying that the data be stored in the data store in a read-write mode, wherein the data stored in the read-write mode is eligible for direct access by a structured query language (SQL) query statement issued to the server cluster.

4. The method of claim 1, wherein determining, by at least one server of the server cluster, that the table data persistence condition has been satisfied comprises:

determining that a data record of the database table satisfies a condition of moving a data record from a memory device to the distributed file system as specified in an eviction setting in the specification.

5. The method of claim 1, wherein determining, by at least one server of the server cluster, that a table data persistence condition has been satisfied comprises:

determining that an amount of data records stored in a memory device satisfies a memory overflow condition in the specification.

6. The method of claim 1, comprising storing a data manipulation language (DML) statement defining a schema of the database table in a metadata file in the data store.

7. The method of claim 1, wherein the removed portion of the queue or the removed portion of data records as log files are readable by a mapper of a Hadoop™

system from the data store.

20. The system of claim 1, the operations comprising storing a data manipulation language (DML) statement defining a schema of the database table in a metadata file in the data store.

8. A non-transitory storage device storing instructions operable to cause a plurality of computers to perform operations comprising:

receiving, by a server cluster, a specification of a database table, the specification including an identifier of a data store on a distributed file system, wherein the server cluster comprises a plurality of servers, each server being a computer program executing on one or more of the plurality of computers, the server cluster being different from the distributed file system;

creating, by the servers in the server cluster, the database table based on the specification, the database table being distributed among each server of the server cluster;

storing, by each server in the server cluster, a history of modification of a portion of the database table distributed to that server as table data in a queue of that server, the history including events of inserting data records into the database table and events of changing data records in the database table, wherein the queue of that server and data records of the database table that are distributed to that server are stored on a storage device of the one or more computers allocated to that server;

determining, by at least one server of the server cluster, that a table data persistence condition has been satisfied on the at least one server;

in response, removing, by the at least one server, at least a portion of the queue from a storage device associated with the at least one server or removing at least a portion of the data records from the storage device associated with the at least one server; and

storing the removed portion of the queue or the removed portion of data records as log files in the data store of the distributed file system.

9. The non-transitory storage device of claim 8, wherein the specification includes a parameter specifying that the table data be stored in the data store in a write-only mode, wherein the data stored in the write-only mode is ineligible for direct access by a structured query language (SQL) query statement issued to the server cluster.

10. The non-transitory storage device of claim 8, wherein the specification includes a parameter specifying that the data be stored in the data store in a read-write mode, wherein the data stored in the read-write mode is eligible for direct access by a structured query language (SQL) query statement issued to the server cluster.

11. The non-transitory storage device of claim 8, wherein determining, by at least one server of the server cluster, that the table data persistence condition has been satisfied comprises:

determining that a data record of the database table satisfies a condition of moving a data record from a memory device to the distributed file system as specified in an eviction setting in the specification.

12. The non-transitory storage device of claim 8, wherein determining, by at least one server of the server cluster, that a table data persistence condition has been satisfied comprises:

determining that an amount of data records stored in a memory device satisfies a memory overflow condition in the specification.

13. The non-transitory storage device of claim 8, the operations comprising storing a data manipulation language (DML) statement defining a schema of the database table in a metadata file in the data store.

14. The non-transitory storage device of claim 8, wherein the removed portion of the queue or the removed portion of data records as log files are readable by a mapper of a Hadoop™

system from the data store.

15. A system comprising:

one or more computers; and

a non-transitory storage device storing instructions operable to cause the one or more computers to perform operations comprising;

receiving, by a server cluster, a specification of a database table, the specification including an identifier of a data store on a distributed file system, wherein the server cluster comprises a plurality of servers, each server being a computer program executing on one or more of the plurality of computers, the server cluster being different from the distributed file system;

creating, by the servers in the server cluster, the database table based on the specification, the database table being distributed among each server of the server cluster;

storing, by each server in the server cluster, a history of modification of a portion of the database table distributed to that server as table data in a queue of that server, the history including events of inserting data records into the database table and events of changing data records in the database table, wherein the queue of that server and data records of the database table that are distributed to that server are stored on a storage device of the one or more computers allocated to that server;

determining, by at least one server of the server cluster, that a table data persistence condition has been satisfied on the at least one server;

in response, removing, by the at least one server, at least a portion of the queue from a storage device associated with the at least one server or removing at least a portion of the data records from the storage device associated with the at least one server; and

storing the removed portion of the queue or the removed portion of data records as log files in the data store of the distributed file system.

16. The system of claim 15, wherein the specification includes a parameter specifying that the table data be stored in the data store in a write-only mode, wherein the data stored in the write-only mode is ineligible for direct access by a structured query language (SQL) query statement issued to the server cluster.

17. The system of claim 15, wherein the specification includes a parameter specifying that the data be stored in the data store in a read-write mode, wherein the data stored in the read-write mode is eligible for direct access by a structured query language (SQL) query statement issued to the server cluster.

18. The system of claim 15, wherein determining, by at least one server of the server cluster, that the table data persistence condition has been satisfied comprises:

determining that a data record of the database table satisfies a condition of moving a data record from a memory device to the distributed file system as specified in an eviction setting in the specification.

19. The system of claim 15, wherein determining, by at least one server of the server cluster, that a table data persistence condition has been satisfied comprises:

determining that an amount of data records stored in a memory device satisfies a memory overflow condition in the specification.

21. The system of claim 15, wherein the removed portion of the queue or the removed portion of data records as log files are readable by a mapper of a Hadoop™

system from the data store.