Systems and methods for managing distributed database deployments

• US 10,713,280 B2
• Filed: 06/20/2017
• Issued: 07/14/2020
• Est. Priority Date: 12/23/2010
• Status: Active Grant

• Current Assignee: MongoDB, Inc.
• Original Assignee: MongoDB, Inc.
• Inventors: Horowitz, Eliot, Morales, John, Mintz, Cory P., Berger, Louisa, Nelson, Cailin Anne
• Primary Examiner(s): Ko, Chae M
• Application Number:
  US15/627,645
• Publication Number:
  US 20170286517A1
• Time in Patent Office: 1,120 Days
• Field of Search: 714 63, 714 13
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0754 by exceeding limits
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data

Abstract

Various aspects provide for implementation of a cloud service for running, monitoring, and maintaining cloud distributed database deployments and in particular examples, provides cloud based services to run, monitor and maintain deployments of the known MongoDB database. Various embodiments provide services, interfaces, and manage provisioning of dedicated servers for the distributed database instances (e.g., MongoDB instances). Further aspects, including providing a database as a cloud service that eliminates the design challenges associated with many distributed database implementations, while allowing the client'"'"'s input on configuration choices in building the database. In some implementations, clients can simply identity a number of database nodes, capability of the nodes, and within minutes have a fully functioning, scalable, replicated, and secure distributed database in the cloud.

Claims

1. A cloud distributed database, the database comprising:

at least one cloud based resource, the at least one cloud based resource including processor and memory;

a database subsystem executing on the at least one cloud based resource, wherein the database subsystem comprises;

at least one replica set comprising a primary node hosting a primary database instance that accepts database write operations from client systems, and at least two secondary nodes that host copies of data of the primary database instance that replicate logged operations from the primary node;

a proxy layer configured to;

control authentication of client systems based on connection strings communicated from the client systems connecting to the at least one cloud resources;

assign a portion of the at least one cloud based resource and operating capability of the database subsystem to respective clients;

manage access to the database application for respective clients based on the client connection strings to enable transparent execution of write and read commands against the database subsystem and respective portions of the database hosted by the replica set, and communicate a result to the client system from the portion of database associated with the client system responsive to client communicated operations; and

a monitoring component, executing within the proxy layer, configured to;

monitor usage metrics for cloud resource utilization and database operation;

wherein the proxy layer is further configured to limit resource utilization by any client to pre-defined thresholds for processor, bandwidth, memory, and storage based on usage metrics from the monitoring component.

2. The system of claim 1, wherein the database subsystem executes a multi-tenant database instance hosted on the at least one replica set.

3. The system of claim 2, wherein the proxy layer manages secure portions of the multi-tenant database instance for each of the portion of the at least one cloud based resource and the operating capability of the database subsystem assigned to respective clients.

4. The system of claim 1, wherein the monitoring component further comprises at least one monitor agent executed on instantiated cloud resources configured to collect the performance metrics for cloud resource utilization.

5. The system of claim 4, wherein the monitoring component further comprising at least one monitor agent executing on database application components configured to collect performance metrics for database operation.

6. The system of claim 5, wherein the database subsystem is further configured to automatically provision additional cloud resources to support the at least one replica set;

and maintain minimum threshold of resource for respective clients.

7. The system of claim 1, wherein the proxy layer is further configured to connect a respective portion of the database subsystem associated a client with an independent complete database instance associated with the client.

8. The system of claim 7, wherein the database subsystem is configured display a management interface configured to manage connected multi-tenant and non-multi-tenant database instances of the client.

9. The system of claim 1, further comprising;

a snapshot component, executed on the at least one cloud resource, configured to;

generate a plurality of snapshots of data stored in the at least one replica set; and

identify a committed snapshot from the plurality of snapshots that is representative of data that has been replicated on a majority of members of the at least one replica set; and

a command processing component configured to read the committed snapshot responsive specification of a read commit option in a client request and generate the result of data matching the read request using the committed snapshot.

10. The system of claim 1, wherein the database subsystem is further configured to disable memory or processor intensive database operations.

11. The system of claim 1, wherein the at least one replica set is configured to:

communicate, via a normal database operation, heartbeat information as metadata within a write operation function; and

automatically recover a primary node in the distributed database system in response to a detected failure of the primary node by an absence of the heartbeat information.

12. A computer implemented method for managing a cloud distributed database, the method comprising:

executing a database subsystem on at least one cloud based resource having memory and processor, wherein the database subsystem comprises at least one replica set including a primary node hosting a primary database instance, and at least two secondary nodes that host copies of data of the primary database instance;

executing, by the database subsystem, write operations at the primary database instance and replicating the write operations to the at least two secondary nodes;

controlling, by a proxy layer executing on the at least one cloud resource, authentication, of client systems based on connection strings communicated from the client systems connecting to the at least one cloud resources;

assigning, by the proxy layer, a portion of the at least one cloud based resource and operating capability of the database subsystem to respective clients;

managing, by the proxy layer, access to the database application for respective clients based on the client connection strings to enable transparent execution of write and read commands against the database subsystem and respective portions of the database hosted by the replica set; and

communicating, by the proxy layer, a result to the client system from the portion of database associated with the client system responsive to client communicated operations; and

monitoring, by a monitor component executed on the at least one cloud resource, usage metrics for cloud resource utilization and database operation; and

limiting, by the proxy layer, resource utilization by any client to pre-defined thresholds for processor, bandwidth, memory, and storage based on usage metrics obtained from the monitoring component.

13. The method of claim 12, wherein the method further comprises executing, on the database subsystem, a multi-tenant database instance hosted on the at least one replica set.

14. The method of claim 13, wherein the method further comprises managing, by the proxy layer, secure portions of the multi-tenant database instance for each portion of the at least one cloud based resource and the operating capability of the database subsystem assigned to respective clients.

15. The method of claim 12, wherein the method further comprises collecting, by the monitoring component, the performance metrics for cloud resource utilization.

16. The method of claim 15, wherein the method further comprises executing at least one monitor agent executing on respective database application components;

and collecting the performance metrics for database operation with respective at least one monitor agent.

17. The method of claim 16, wherein the method further comprises:

provisioning, automatically, additional cloud resources to support the at least one replica set; and

maintaining a minimum threshold of resource for respective clients.

18. The method of claim 12, wherein the method further comprises connecting, by the proxy layer, a respective portion of the database subsystem associated a client with an independent database instance associated with the client.

19. The method of claim 18, wherein the method further comprises displaying, a management interface configured to manage connected multi-tenant and non-multi-tenant database instances of the client.

20. The method of claim 12, wherein the method further comprises disabling memory or processor intensive database operations.

21. The method claim 12, wherein the method further comprises:

generating, by a snapshot component executed on the at least one cloud resource, a plurality of snapshots of data stored in the at least one replica set;

identify, by the snapshot component, a committed snapshot from the plurality of snapshots that is representative of data that has been replicated on a majority of members of the at least one replica set; and

reading, data from the committed snapshot responsive to specification of a read commit option in a client request and generating the result of data matching the read request using the committed snapshot.

22. The method of claim 12, wherein the method further comprises:

communicating, via a normal database operation, heartbeat information as metadata within a write operation function; and

automatically recovering a primary node in the distributed database system in response to a detected failure of the primary node by an absence of the heartbeat information.