Dynamic process recovery in a distributed environment

• US 20060277538A1
• Filed: 06/02/2005
• Published: 12/07/2006
• Est. Priority Date: 06/02/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Graber, Lee B., Saha, Sanjib, Sujanani, Sumitra
• Granted Patent: US 7,783,742 B2
• Time in Patent Office: Days
• US Class Current: 717/168
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating

Abstract

Methods for providing dynamic process recovery that supports recovery of event driven services. The present invention addresses situations where a service-hosting process fails as well as all of the services which were running in that process. To recover from this scenario, the present invention restarts the services a different process from a last know good state. The present invention also provides for situations where a process is shut down in a controlled manner, but the services running within it are still able to drive. Here again, the services are restarted in a secondary process. The recovery modal described here, therefore, advantageously addresses a problem in the design of a distributed event-driven environment.

Claims

1. A method of process recovery in a distributed environment, comprising:

detecting a process heartbeat associated with a first instance of a process, said process having services;

determining that said process has failed if said heartbeat was not received at a predetermined next heartbeat time;

releasing resources associated with said process;

restarting a second instance of said process and said services from a last known good state of said first instance of said process.

2. The method of claim 1, further comprising storing information regarding said process heartbeat in a database table;

and

scheduling a database agent job to check a current time against said next heartbeat time.

3. The method of claim 1, said restarting said second instance of said process further comprising:

determining if messages are marked for services within the said process; and

if there are no messages marked for services of the said process, providing a restart message to restart the said services within said second instance of said process.

4. The method of claim 3, further comprising providing said restart message with no content or properties.

5. The method of claim 1, said restarting said services in the second instance of said process further comprising:

determining if said service has resources which are not persistent or if said service does not need to be recovered;

deleting entries in a database table associated with said service; and

not restarting said service in the second instance of said process.

6. The method of claim 1, further comprising ascertaining if said process was incorrectly determined as failed by comparing an identifier associated with said first instance of said process with an identifier in a database table.

7. The method of claim 1, further comprising:

determining if said first instance of said process is being shutdown in a controlled manner;

determining if said services running in said process are active; and

removing said process from memory.

8. The method of claim 7, further comprising:

determining if said service will restart; and

if not, providing a restart message to said service to restart said second instance of said process.

9. A computer readable medium having computer-readable instructions stored thereon for process recovery in a distributed environment, said computer-readable instructions performing the method comprising:

detecting a process heartbeat associated with a first instance of a process, said process having services;

determining that said process has failed if said heartbeat was not received at a predetermined next heartbeat time;

releasing resources associated with said process;

restarting a second instance of said process and said services from a last known good state of said first instance of said process.

10. The computer readable medium of claim 9, further comprising instructions for storing information regarding said process heartbeat in a database table;

and

scheduling a database agent job to check a current time against said next heartbeat time.

11. The computer readable medium of claim 9, said restarting said second instance of said process further comprising instructions for:

determining if messages are marked for the said services of the said process; and

if there are no messages marked for said services, providing a restart message to restart said second instance of said process.

12. The computer readable medium of claim 11, further comprising instructions for providing said restart message with no content or properties.

13. The computer readable medium of claim 9, said restarting second instance of said process further comprising instructions for:

determining if said service has resources which are not persistent or if said process does not need to be recovered;

deleting entries in a database table associated with said service; and

not restarting the service in the said second instance of said process.

14. The computer readable medium of claim 9, further comprising instructions for ascertaining if said process was incorrectly determined as failed by comparing an identifier associated with said first instance of said process with an identifier in a database table.

15. The computer readable medium of claim 9, further comprising instructions for:

determining if said first instance of said process is being shutdown in a controller manner;

determining if said services running in said process are active; and

removing said process from memory.

16. The computer readable medium of claim 15, further comprising instructions for:

determining if said service of the said process will restart; and

if not, providing a restart message to said service to restart in said second instance of said process.

17. A distributed computing environment, comprising:

a first server running a process, said process having services;

a second server having a backup process, said backup process is capable of running services if the first server is unavailable; and

a database server that tracks a heartbeat from said first instance of said process,

wherein if it is determined said process has failed because said heartbeat was not received at a predetermined next heartbeat time, said database server releases resources associated with said process and restarts said services in said backup process from a last known good state of said first instance of said process.

18. The distributed computing environment of claim 17, wherein said database server determines if messages are marked for said services of the said process, and if there are no messages marked for said services process, providing a restart message to said backup process.

19. The distributed computing environment of claim 17, wherein said database server determines if said process has resources which are not persistent, and if so, deletes entries in a database table associated with said process.

20. The distributed computing environment of claim 17, wherein if said first instance of said process is being shutdown in a controller manner, said database process determines if said services running in said process are active and if said process will restart, and

wherein if said services running in said process are active and said process will not restart, said database server provides a restart message to said service process to restart it in said backup process.