Distributed database systems and methods with pluggable storage engines

• US 10,262,050 B2
• Filed: 01/11/2016
• Issued: 04/16/2019
• Est. Priority Date: 09/25/2015
• Status: Active Grant

• Current Assignee: MongoDB, Inc.
• Original Assignee: MongoDB, Inc.
• Inventors: Bostic, Keith, Cahill, Michael, Horowitz, Eliot, Pasette, Dan, Stearn, Mathias, Bosch, Geert
• Primary Examiner(s): Mobin, Hasanul
• Application Number:
  US14/992,225
• Publication Number:
  US 20170091327A1
• Time in Patent Office: 1,191 Days
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

According to one aspect, methods and systems are provided for selectively employing storage engines in a distributed database environment. The methods and systems can include a processor configured to execute a plurality of system components, wherein the system components comprise an operation prediction component configured to determine an expected set of operations to be performed on a portion of the database; a data format selection component configured to select, based on at least one characteristic of the expected set of operations, a data format for the portion of the database; and at least one storage engine for writing the portion of the database in the selected data format.

Claims

1. A database system comprising:

a database;

at least one processor configured to execute a plurality of system components, wherein the system components comprise;

a plurality of storage engines configured to store data in the database in a plurality of different data formats;

a storage application program interface (API) configured to receive requests from a client to perform operations on the database;

an operation prediction component configured to determine an expected set of operations to be performed on a portion of the database; and

a data format selection component configured to select, based on at least one characteristic of the expected set of operations, a data format of the plurality of different data formats for storing data in the portion of the database, wherein the selected data format optimizes execution of the expected set of operations on the portion of the database relative to another one of the plurality of data formats;

wherein the storage API is configured to trigger execution of at least one of the plurality of storage engines responsive to a request to perform a write operation, the at least one storage engine configured to write data to the portion of the database in the selected data format.

2. The system of claim 1, wherein the operation prediction component is further configured to access information about a past set of operations for a first time period, and predict, based on the past set of operations for the first time period, an expected set of operations to be performed on the portion of the database during a second time period.

3. The system of claim 1, wherein the operation prediction component is further configured to determine the expected set of operations to be performed on the portion of the database by identifying a data structure for data to be stored in the portion of the database.

4. The system of claim 1, wherein the characteristic of the expected set of operations comprises a ratio of read operations to write operations.

5. The system of claim 1, wherein the data format is a row-store format.

6. The system of claim 1, wherein the data format is a column-store format.

7. The system of claim 1, wherein the characteristic of the expected set of operations comprises a determination that sequential operations are likely to be performed on a first storage location and a second storage location nearby the first storage location.

8. The system of claim 1, wherein the characteristic of the expected set of operations comprises a ratio of write operations to read operations.

9. The system of claim 8, wherein the data format is a log-structured merge-tree format.

10. The system of claim 1, wherein the characteristic of the expected set of operations comprises a requirement to update less than all of the fields in a plurality of records stored in the database, and wherein the data format is a column-store format.

11. A method of performing operations in a computer database, comprising steps of:

storing, by a plurality of storage engines, data in a database in a plurality of different data formats;

receiving, by a storage API, requests from a client to perform operations on the database;

determining an expected set of operations to be performed on a portion of the database;

selecting, based on at least one characteristic of the expected set of operations, a data format of the plurality of data formats for storing data in the portion of the database, wherein the selected data format optimizes execution of the expected set of operations on the portion of the database relative to another one of the plurality of data formats;

storing the selected data format in a configuration metadata component of the computer database;

receiving, by the storage API, a request to perform a write operation;

triggering, by the storage API, execution of at least one of the plurality of storage engines responsive to receiving a request to perform a write operation; and

writing, by the at least one storage engine, data to the portion of the database in the selected data format.

12. The method of claim 11, wherein determining the expected set of operations to be performed on the portion of the database comprises:

accessing information about a past set of operations for a first time period; and

predicting, based on the past set of operations for the first time period, an expected set of operations to be performed on the portion of the database during a second time period.

13. The method of claim 11, wherein determining the expected set of operations to be performed on the portion of the database comprises identifying a data structure for data to be stored in the portion of the database.

14. The method of claim 11, wherein the characteristic of the expected set of operations comprises a ratio of read operations to write operations.

15. The method of claim 14, wherein the first data format is a row-store format.

16. The method of claim 14, wherein the first data format is a column-store format.

17. The method of claim 11, wherein the characteristic of the expected set of operations comprises a determination that sequential operations are likely to be performed on a first storage location and a second storage location nearby the first storage location.

18. The method of claim 11, wherein the characteristic of the expected set of operations comprises high ratio of write operations to read operations.

19. The method of claim 18, wherein the second data format is a log-structured merge-tree format.

20. The method of claim 11, wherein the first characteristic of the expected set of operations is a requirement to update less than all of the fields in a plurality of records stored in the database, and wherein the first data format is a column-store format.