DISTRIBUTED DATABASE SYSTEMS AND METHODS WITH PLUGGABLE STORAGE ENGINES

• US 20170091327A1
• Filed: 01/11/2016
• Published: 03/30/2017
• Est. Priority Date: 09/25/2015
• Status: Active Grant

• Current Assignee: MongoDB, Inc.
• Original Assignee: MongoDB, Inc.
• Inventors: Bostic, Keith, Cahill, Michael, Pasette, Dan, Bosch, Geert, Horowitz, Eliot, Stearn, Mathias
• Granted Patent: US 10,262,050 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

According to one aspect, methods and systems are provided for selectively employing storage engines in a distributed database environment. The methods and systems can include a processor configured to execute a plurality of system components, wherein the system components comprise an operation prediction component configured to determine an expected set of operations to be performed on a portion of the database; a data format selection component configured to select, based on at least one characteristic of the expected set of operations, a data format for the portion of the database; and at least one storage engine for writing the portion of the database in the selected data format.

Claims

1. A database system comprising:

at least one processor configured to execute a plurality of system components, wherein the system components comprise;

an operation prediction component configured to determine an expected set of operations to be performed on a portion of the database;

a data format selection component configured to select, based on at least one characteristic of the expected set of operations, a data format for the portion of the database; and

at least one storage engine for writing the portion of the database in the selected data format.

2. The system of claim 1, wherein the operation prediction component is further configured to access information about a past set of operations for a first time period, and predict, based on the past set of operations for the first time period, an expected set of operations to be performed on the portion of the database during a second time period.

3. The system of claim 1, wherein the operation prediction component is further configured to determine the expected set of operations to be performed on the portion of the database by identifying a data structure for data to be stored in the portion of the database.

4. The system of claim 1, wherein the characteristic of the expected set of operations is a relatively high ratio of read operations to write operations.

5. The system of claim 1, wherein the data format is a row-store format.

6. The system of claim 1, wherein the data format is a column-store format.

7. The system of claim 1, wherein the characteristic of the expected set of operations is a determination that sequential operations are likely to be performed on a first storage location and a second storage location nearby the first storage location.

8. The system of claim 1, wherein the characteristic of the expected set of operations is a relatively high ratio of write operations to read operations.

9. The system of claim 8, wherein the data format is a log-sequence merge format.

10. The system of claim 1, wherein the characteristic of the expected set of operations is a requirement to update less than all of the fields in a plurality of records stored in the database, and wherein the data format is a column-store format.

11. A method of performing operations in a computer database, comprising steps of:

determining, by a computer system, an expected set of operations to be performed on a portion of a database;

selecting, based on at least one characteristic of the expected set of operations, a data format for the portion of the database;

storing the selected data format in a configuration metadata component of the computer database; and

writing data to the portion of the database in the selected data format.

12. The method of claim 11, wherein determining the expected set of operations to be performed on the portion of the database comprises:

accessing information about a past set of operations for a first time period; and

predicting, based on the past set of operations for the first time period, an expected set of operations to be performed on the portion of the database during a second time period.

13. The method of claim 11, wherein determining the expected set of operations to be performed on the portion of the database comprises identifying a data structure for data to be stored in the portion of the database.

14. The method of claim 11, wherein the characteristic of the expected set of operations is a relatively high ratio of read operations to write operations.

15. The method of claim 14, wherein the first data format is a row-store format.

16. The method of claim 14, wherein the first data format is a column-store format.

17. The method of claim 11, wherein the characteristic of the expected set of operations is a determination that sequential operations are likely to be performed on a first storage location and a second storage location nearby the first storage location.

18. The method of claim 11, wherein the characteristic of the expected set of operations is a relatively high ratio of write operations to read operations.

19. The method of claim 18, wherein the second data format is a log-sequence merge format.

20. The method of claim 11, wherein the first characteristic of the expected set of operations is a requirement to update less than all of the fields in a plurality of records stored in the database, and wherein the first data format is a column-store format.

21. A method of performing operations in a computer database, comprising steps of:

presenting, in a user interface of a computer system, a plurality of data format options for a portion of a database;

receiving, from the user interface, a user selection of a data format for the portion of the database;

storing the data format selection as configuration metadata for the database;

responsive to the data format selection indicating a first data format, activating a first storage engine to store the portion of the database in the first data format; and

responsive to the data format selection indicating a second data format, activating a second storage engine to store the portion of the database in the second data format.

22. The method of claim 21, wherein the first data format is a row-store format.

23. The method of claim 21, wherein the first data format is a column-store format.

24. The method of claim 21, wherein the second data format is a log-sequence merge format.

25. A method of performing operations in a computer database, comprising steps of:

receiving, from a computer application, a request to perform a write operation, wherein the request does not specify a data storage format;

selecting, by a computer system, a data storage format from a group consisting of at least a first data storage format and a second data storage format;

responsive to a selection of the first data storage format, performing the write operation using a first data storage engine; and

responsive to a selection of the second data storage format, performing the write operation using a second data storage engine.

26. A database system for storing data in an optimal format, the system comprising:

an application programming interface configured to receive, from a computer system, a request to perform a write operation, wherein the request does not specify a data storage format;

at least one storage component configured to store a plurality of data records;

a first storage engine configured to store the plurality of data records in a first format;

a second storage engine configured to store the plurality of data records in a second format; and

a storage engine selector for selectively executing one of the first storage engine or the second storage engine to perform the write operation.

27. The database system of claim 26, further comprising:

a database monitor configured to track performance information about the database system; and

a memory configured to store analytics data comprising performance information tracked by the database monitor.

28. The database system of claim 26, further comprising:

a configuration database adapted to stored configuration metadata about the database, the configuration metadata including at least one of an association between a storage engine and one of the at least one storage components.

29. A database system for storing data in an optimal format, the system comprising:

an application programming interface configured to receive, from a computer system, a request to perform a write operation, wherein the request does not specify a data storage format;

a replica set comprising a primary node having a first storage component and a secondary node having a second storage component, the first storage component and the second storage component configured to store a plurality of records;

a first storage engine configured to store the plurality of data records in a first format in the first storage component; and

a second storage engine configured to store the plurality of data records in a second format in the second storage component.

30. The database system of claim 29, further comprising a storage engine selector for selectively executing one of the first storage engine or the second storage engine to perform the write operation.