Automated self-healing database system and method for implementing the same

• US 10,409,697 B2
• Filed: 02/23/2017
• Issued: 09/10/2019
• Est. Priority Date: 02/23/2017
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Chen, Bohan, Tam, Donald, Collins, Jesse, Qiao, Sheng, Liu, Ted, Yu, Youngjin
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US15/440,810
• Publication Number:
  US 20180239677A1
• Time in Patent Office: 929 Days
• CPC Class Codes: G06F 11/1666 where the redundant component is memory or memory area
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/2308 Concurrency control transaction processing G06F9/466
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency ensuring consistency in mirrored systems G06F11/2064

Abstract

An automated self-healing database system is provided that includes a primary database, an application server that writes data to the primary database and reads data from the primary database when it is available, a standby database that is a replica of the primary database, and a self-healing module (SHM). The SHM can automatically detect unavailability of the primary database, and if the standby database is available, the SHM can automatically enable the standby database as readable and writable, assign the standby database the role of primary database at the primary site to start replicating data to other standby databases, and use a pre-established connection between the application server and the standby database to allow the application server to read data from the standby database and to write data to the standby database resulting in the standby database assuming role of the primary database at the primary site.

Claims

1. A method in a database system comprising a primary site, wherein the primary site further comprises an application server that writes data to a primary database and reads data from the primary database when the primary database is available, the method comprising:

automatically detecting, by a self-healing module, unavailability of the primary database;

determining, by the self-healing module, whether a standby database of the primary site is available, wherein the standby database is a replica of the primary database;

temporarily allowing the application server to have read-only access from the standby database at the primary site when the self-healing module determines that the standby database is available at the primary site so that the application server can still read data from the standby database and satisfy read-only requests while the primary database is down and it is being determined whether the primary database will recover; and

automatically initiating a failover processing sequence by the self-healing module when the self-healing module detects that the primary database is unavailable and determines that the standby database is available, wherein the failover processing sequence comprises;

automatically enabling the standby database as readable and writable and assigning the standby database the role of primary database at the primary site to start replicating data to other standby databases; and

after a failover role transition, using a pre-established connection between the application server and the standby database to allow the application server to read data from the standby database and to write data to the standby database resulting in the standby database assuming role of the primary database at the primary site, wherein no new connection establishment is needed due to the pre-established connection when the fail over role transition happens.

2. The method according to claim 1, further comprising:

periodically capturing snapshots of data stored at the standby database; and

storing the snapshots of data in a snapshot storage system comprising separate storage hardware that is not implemented at the primary site.

3. The method according to claim 2, wherein the standby database is a first standby database, and, wherein the failover processing sequence further comprises:

automatically creating, at the self-healing module, a new database using most recent snapshot data from the primary database that is stored at a snapshot storage system; and

automatically adding, at the self-healing module, the new database as a second standby database at the primary site to restore fully capacity.

4. The method according to claim 1, wherein the primary database is a first primary database, and wherein the database system further comprises a disaster recover site comprising a second primary database, and the method further comprising:

directing, via a load balancer, traffic from user systems to an application server at the disaster recovery site that has access to the second primary database when all of the databases of the primary site have failed and no standby databases are available at the primary site.

5. The method according to claim 1, further comprising:

determining, at the self-healing module prior to automatically initiating the failover processing sequence, whether the primary database at the primary site has recovered within a recovery time when the self-healing module determines that the standby database is available at the primary site; and

automatically initiating, at the self-healing module, the failover processing sequence when the primary database at the primary site has not recovered within the recovery time.

6. The method according to claim 5, further comprising:

automatically re-starting the primary database when the self-healing module determines that the primary database at the primary site has recovered within the recovery time, wherein applications at the application server continue to use the primary database from the primary site as the primary database.

7. A method in a database system comprising a primary site, wherein the primary site further comprises an application server that writes data to a primary database and reads data from the primary database when the primary database is available, the method comprising:

automatically detecting, by a self-healing module, unavailability of the primary database, wherein the primary database comprises a first database server and a first storage sub-system;

determining, by the self-healing module, whether a standby database of the primary site is available, wherein the standby database is a replica of the primary database, wherein the standby database comprises a second database server and a second storage sub-system; and

automatically initiating a failover processing sequence by the self-healing module when the self-healing module detects that the primary database is unavailable and determines that the standby database is available, wherein the failover processing sequence comprises;

before automatically enabling the standby database, automatically attaching the first storage sub-system from the first database server to the second database server for recovery;

automatically enabling the standby database as readable and writable and assigning the standby database the role of primary database at the primary site to start replicating data to other standby databases; and

after a failover role transition, using a pre-established connection between the application server and the standby database to allow the application server to read data from the standby database and to write data to the standby database resulting in the standby database assuming role of the primary database at the primary site, wherein no new connection establishment is needed due to the pre-established connection when the fail over role transition happens.

8. A computing system comprising a processor and a memory, wherein the memory comprises computer-executable instructions that are capable of causing the computing system to:

automatically detect unavailability of a primary database at a primary site, wherein the primary site further comprises an application server that writes data to the primary database and reads data from the primary database when the primary database is available;

determine whether a standby database is still available at the primary site, wherein the standby database is a replica of the primary database;

temporarily allow the application server to have read-only access from the standby database at the primary site when the self-healing module determines that the standby database is available at the primary site so that the application server can still read data from the standby database and satisfy read-only requests while the primary database is down and it is being determined whether the primary database will recover; and

automatically initiate a failover processing sequence when the primary database has been detected to be unavailable and the standby database has been determined to be available, wherein the failover processing sequence comprises;

automatically enabling the standby database as readable and writable and assigning the standby database the role of primary database at the primary site to start replicating data to other standby databases; and

after a failover role transition, using a pre-established connection between the application server and the standby database to allow the application server to read data from the standby database and to write data to the standby database resulting in the standby database assuming role of the primary database at the primary site, wherein no new connection establishment is needed due to the pre-established connection when the fail over role transition happens.

9. The computing system of claim 8, wherein the computer-executable instructions are further capable of causing the computing system to:

periodically capture snapshots of data stored at the standby database, wherein the standby database is a first standby database;

store the snapshots of data in a snapshot storage system comprising separate storage hardware that is not implemented at the primary site;

automatically create a new database using most recent snapshot data from the primary database that is stored at a snapshot storage system; and

automatically add the new database as a second standby database at the primary site to restore fully capacity.

10. A computing system comprising a processor and a memory, wherein the memory comprises computer-executable instructions that are capable of causing the computing system to:

automatically detect unavailability of a primary database at a primary site, wherein the primary site further comprises an application server that writes data to the primary database and reads data from the primary database when the primary database is available, wherein the primary database comprises a first database server and a first storage sub-system;

determine whether a standby database is still available at the primary site, wherein the standby database is a replica of the primary database, wherein the standby database comprises a second database server and a second storage sub-system; and

automatically initiate a failover processing sequence when the primary database has been detected to be unavailable and the standby database has been determined to be available, wherein the failover processing sequence comprises;

before automatically enabling the standby database, automatically attaching the first storage sub-system from the first database server to the second database server for recovery before automatically enabling;

automatically enabling the standby database as readable and writable and assigning the standby database the role of primary database at the primary site to start replicating data to other standby databases; and

after a failover role transition, using a pre-established connection between the application server and the standby database to allow the application server to read data from the standby database and to write data to the standby database resulting in the standby database assuming role of the primary database at the primary site, wherein no new connection establishment is needed due to the pre-established connection when the fail over role transition happens.

11. An automated self-healing database system, comprising:

a primary site, comprising;

a primary database;

an application server that writes data to the primary database and reads data from the primary database when the primary database is available;

a standby database that is a replica of the primary database; and

a self-healing module configured to;

automatically detect unavailability of the primary database;

determine whether the standby database of the primary site is available;

temporarily allow the application server to have read-only access from the standby database at the primary site when the self-healing module determines that the standby database is available at the primary site so that the application server can still read data from the standby database and satisfy read-only requests while the primary database is down and it is being determined whether the primary database will recover; and

automatically initiate a failover processing sequence when the self-healing module detects that the primary database is unavailable and determines that the standby database of the primary site is available, wherein the failover processing sequence comprises;

automatically enabling the standby database as readable and writable and assigning the standby database the role of primary database at the primary site to start replicating data to other standby databases; and

after a failover role transition, using a pre-established connection between the application server and the standby database to allow the application server to read data from the standby database and to write data to the standby database resulting in the standby database assuming role of the primary database at the primary site, wherein no new connection establishment is needed due to the pre-established connection when the fail over role transition happens.

12. The automated self-healing database system according to claim 11, wherein the self-healing module is further configured to periodically capture snapshots of data stored at the standby database, wherein the automated self-healing database system further comprises:

a snapshot storage system comprising separate storage hardware that is remotely located and not implemented at the primary site, wherein the snapshot storage system is configured to store the snapshots of data.

13. The automated self-healing database system according to claim 12, wherein the standby database is a first standby database, and wherein the self-healing module is further configured to, as part of the failover processing sequence, automatically create a new database using most recent snapshot data from the primary database, that is stored at a snapshot storage system, and automatically add the new database as a second standby database at the primary site to restore fully capacity.

14. The automated self-healing database system according to claim 11, wherein the primary database is a first primary database, and wherein the automated self-healing database system further comprises:

a load balancer; and

a disaster recover site comprising an application server and a second primary database, and

wherein the load balancer is configured to direct traffic from user systems to the application server at the disaster recover site that has access to the second primary database when all of the databases of the primary site have failed and no standby databases are available at the primary site.

15. The automated self-healing database system according to claim 11, wherein the self-healing module is further configured to:

determine, prior to automatically initiating the failover processing sequence, whether the primary database at the primary site has recovered within a recovery time when the self-healing module determines that the standby database is available at the primary site; and

automatically initiate the failover processing sequence when the primary database at the primary site has not recovered within the recovery time.

16. The automated self-healing database system according to claim 15, wherein the self-healing module is further configured to:

automatically re-start the primary database when the self-healing module determines that the primary database at the primary site has recovered within the recovery time, wherein applications at the application server continue to use the primary database from the primary site as the primary database.

17. An automated self-healing database system, comprising:

a primary site, comprising;

a primary database;

an application server that writes data to the primary database and reads data from the primary database when the primary database is available;

a standby database that is a replica of the primary database; and

a self-healing module configured to;

automatically detect unavailability of the primary database;

determine whether the standby database of the primary site is available; and

automatically initiate a failover processing sequence when the self-healing module detects that the primary database is unavailable and determines that the standby database of the primary site is available, wherein the failover processing sequence comprises;

before automatically enabling the standby database, automatically attaching the first storage sub-system from the first database server to the second database server for recovery;

automatically enabling the standby database as readable and writable and assigning the standby database the role of primary database at the primary site to start replicating data to other standby databases; and

after a failover role transition, using a pre-established connection between the application server and the standby database to allow the application server to read data from the standby database and to write data to the standby database resulting in the standby database assuming role of the primary database at the primary site, wherein no new connection establishment is needed due to the pre-established connection when the fail over role transition happens.