Fault tolerance for complex distributed computing operations

• US 10,162,708 B2
• Filed: 07/21/2015
• Issued: 12/25/2018
• Est. Priority Date: 01/13/2012
• Status: Active Grant

• Current Assignee: NetSuite Inc. (Oracle Corporation)
• Original Assignee: NetSuite Inc. (Oracle Corporation)
• Inventors: Parra, Ivan Omar, Williams, Douglas H.
• Primary Examiner(s): Duncan, Marc
• Application Number:
  US14/805,219
• Publication Number:
  US 20160019123A1
• Time in Patent Office: 1,253 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/1482 by means of middleware or OS functionality
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/51 Discovery or management thereof, e.g. service location protocol [SLP] or web services

Abstract

A method for enabling a distributed computing system to tolerate system faults during the execution of a client process. The method includes instantiating an execution environment relating to the client process; executing instructions within the execution environment, the instructions causing the execution environment to issue further instructions to the distributing computing system, the further instructions relating to actions to be performed with respect to data stored on the distributed computing system. An object interface proxy receives the further instructions and monitors the received to determine if the execution environment is in a desired save-state condition; and, if so, save a current state of the execution environment in a data store.

Claims

1. A method, comprising:

instantiating a multi-tenant distributed computing service environment configured to host a plurality of tenant execution environments executing thereon,

instantiating a tenant execution environment within the multi-tenant distributed computing service environment relating to at least one tenant process;

executing a first set of instructions within the multi-tenant distributed computing service environment, the first set of instructions causing a first save state interface element of the multi-tenant distributed computing service environment to be activated;

as a result of the first save state interface element being activated, causing data corresponding to a first process implemented within the multi-tenant distributed computing service environment to be saved in a first data store;

in response to a monitored call to initiate a process, or based on an application message between the tenant execution environment and a business object layer, executing a second set of instructions within the tenant execution environment, the second set of instructions causing the tenant execution environment to issue a second save state instruction corresponding to the tenant execution environment to the multi-tenant distributed computing service environment; and

receiving the second save state instruction by an object interface proxy executing in the multi-tenant distributed computing service environment which causes the object interface proxy to save a current state of the tenant execution environment in a second data store.

2. The method of claim 1, wherein the object interface proxy functions as an application programming interface for the multi-tenant distributed computing service environment.

3. The method of claim 1, further comprising:

detecting a fault;

determining the fault has occurred within the tenant execution environment;

instantiating a new tenant execution environment relating to the tenant process;

receiving a restore-state instruction by the object interface proxy indicating the tenant process has experienced the fault; and

in response to the restore-state instruction, retrieving the current state of the tenant execution environment from the second data store and placing the new tenant execution environment in the retrieved current state.

4. A system, comprising:

a multi-tenant distributed computing system configured to tolerate system faults in more than one execution environment, the multi-tenant distributed computing system further configured to instantiate a first execution environment configured to execute instructions on at least one software object in accordance with at least one multi-tenant distributed computing system process and configured to instantiate a second execution environment within the first execution environment and configured to execute instructions on at least one software object in accordance with at least one tenant process;

a plurality of data stores comprising a first data store for storing data relating to a state of a process implemented in the first execution environment and a second data store for storing data relating to a state of the second execution environment;

an object interface proxy configured to receive a message corresponding to the process from the first execution environment and save a current state of the process within first environment to the first data store, wherein the object interface proxy is further configured to receive another message and cause a current state of the second execution environment to be saved to the second data store.

5. The system of claim 4, wherein the object interface proxy is configured as an application programming interface for the multi-tenant distributed computing system.

6. The system of claim 4, wherein the object interface proxy is further configured to receive a restore-state instruction from the second execution environment, the receipt of which causes the object interface proxy to place the second execution environment in the current state of the second execution environment.

7. The system of claim 4, wherein the object interface proxy is further configured to issue a save state instruction in response to a monitored call.

8. The system of claim 4, wherein the object interface proxy is further configured to issue a save state instruction based on an application message between the second environment and a business object layer.

9. The system of claim 4, wherein the second fault tolerance module is further configured to determine a frequency of data being manipulated by the second environment and issue a save state instruction in response to the determined frequency.

10. A method, comprising:

hosting a multi-tenant application on a distributed computing system, the multi-tenant application being configured to host a plurality of tenant execution environments;

instantiating a tenant execution environment on the distributed computing system relating to at least one tenant process;

activating a first save state element corresponding to a process of the multi-tenant application based on a type of call monitored by an object interface proxy of the multi-tenant application;

activating a second save state element corresponding to the tenant execution environment based on an application message between the tenant execution environment and a business object layer within the multi-tenant application; and

storing a current state of the process of the multi-tenant application in a first data store in response to activation of the first save state element and storing a current state of the tenant execution environment in a second data store in response to activation of the second save state element.

11. The method of claim 10, further comprising receiving an instruction to activate the second save state element by the object interface proxy executing in the multi-tenant application, which causes the object interface proxy to save the current state of the tenant execution environment in the data store.

12. The method of claim 10, wherein the type of call monitored by the multi-tenant application comprises a call corresponding to a data-write operation.

13. The method of claim 10, wherein the type of call monitored by the multi-tenant application comprises a call corresponding to a data-read operation.

14. The method of claim 10, further comprising:

detecting a fault;

determining the fault has occurred within the tenant execution environment;

instantiating a new tenant execution environment;

receiving a restore-state instruction indicating the tenant execution environment has experienced the fault; and

in response to the restore-state instruction, retrieving the current state of the tenant execution environment from the second data store and placing the new tenant execution environment in the retrieved current state.