Method and apparatus for run-time deconfiguration of a processor in a symmetrical multi-processing system

• US 6,516,429 B1
• Filed: 11/04/1999
• Issued: 02/04/2003
• Est. Priority Date: 11/04/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: OʼQuin, John Thomas II, Kitamorn, Alongkorn, McLaughlin, Charles Andrew, Bossen, Douglas Craig
• Primary Examiner(s): Beausoleil, Robert
• Assistant Examiner(s): Wilson, Yolanda L.
• Application Number:
  US09/434,767
• Time in Patent Office: 1,188 Days
• Field of Search: 714/47, 714/10, 714/48, 714/13, 714/718, 714/31, 714/11
• US Class Current: 714/47.2
• CPC Class Codes: G06F 11/004 Error avoidance G06F11/07 and subgroups take precedence
• CPC Class Codes: G06F 11/0724 in a multiprocessor or a multi-core unit multiprocessors per se G06F15/80
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2028 eliminating a faulty processor or activating a spare
• CPC Class Codes: G06F 11/2035 without idle spare hardware
• CPC Class Codes: G06F 2212/1032 Reliability improvement, data loss prevention, degraded operation etc

Abstract

A method and apparatus in a multiprocessor data processing system for managing a plurality of processors. Monitoring for recoverable errors in a set of processors is performed. Responsive to detecting a recoverable error for a processor in the set of processors, a determination is made as to whether the recoverable error indicates a trend towards an unrecoverable error. Responsive to a determination that the recoverable error indicates a trend towards an unrecoverable error, actions are initiated to stop the processor.

Claims

1. A method in a multiprocessor data processing system for managing a plurality of processors, the method comprising:

monitoring for recoverable errors in a set of processors;

responsive to detecting a recoverable error for a processor in the set of processors, determining whether the recoverable error indicates a trend towards an unrecoverable error; and

responsive to a determination that the recoverable error indicates a trend towards an unrecoverable error, initiating actions to stop the processor.

2. The method of claim 1 further comprising:

responsive to detecting a recoverable error for a processor in the set of processors, maintaining a record of the recoverable error.

3. The method of claim 2, wherein the determining step comprises:

determining whether the record indicates a pattern of recoverable errors trending towards a hard error.

4. The method of claim 3, wherein the pattern is a number of errors exceeding a threshold.

5. The method of claim 1, wherein the initiating step includes:

initiating a migration of processes from the processor to other processors in the multiprocessor data processing system.

6. The method of claim 1, wherein the initiating step includes:

initiating a migration of interrupt handlers from the processor to other processors in the multiprocessor data processing system.

7. The method of claim 6, wherein the migration of interrupt handlers are handled by an operating system for the multiprocessor data processing system.

8. The method of claim 1, wherein the initiating step includes:

stopping dispatch tasks on the processor.

9. The method of claim 1, wherein the initiating step includes:

stopping inter-processor interrupts to the processor.

10. The method of claim 1, wherein the initiating step includes:

storing data from a cache used by the processor in a memory.

11. The method of claim 1 further comprising:

maintaining a record of the processor, wherein the processors in the record are left out of the system configuration on a system boot process.

12. The method of claim 1, wherein the monitoring, determining, and initiating steps are located in firmware for the multiprocessor data processing system.

13. A method in a multiprocessor data processing system for managing a plurality of processors, the method comprising:

monitoring the plurality of processors for errors;

responsive to detecting a recoverable error localized to a processor within the plurality of processors, adding the recoverable error to a record of errors for the processor;

responsive to adding the recoverable error to the record, determining whether the record indicates a trend towards an occurrence of an unrecoverable error; and

responsive to determining that the record indicates a trend towards an occurrence of an unrecoverable error, identifying the processor as a processor that should be removed.

14. The method of claim 13 further comprising:

sending the identification of the processor to an operating system for the multiprocessor data processing system to initiate actions to migrate processes away from the processors to other processors within the plurality of processors.

15. The method of claim 13 further comprising:

transitioning the multiprocessor data processing system to manage the plurality of processors minus the processor.

16. The method of claim 15 further comprising:

moving data from a cache used by the processor to a memory in the multiprocessor data processing system.

17. The method of claim 15 further comprising:

removing the processors from a system configuration for the multiprocessor data processing system during a next system boot of the multiprocessor data processing system.

18. The method of claim 13, wherein the multiprocessor data processing system is a symmetrical multiprocessor system.

19. The method of claim 13, wherein the steps of monitoring, adding, determining, and identifying are performed by a run time system firmware and a service processor in the multiprocessor data processing system.

20. A data processing system comprising:

a set of processors;

a run time system firmware; and

a service processor connected to the set of processors, wherein the run time system firmware and the service processor monitor the set of processors for a selected type of fault in which the selected type of fault is an indication of a processor degradation, identify a processor from the set of processors causing the selected type of fault, and dynamically initiate actions to stop the processor in response to detecting the selected type of fault.

21. The data processing system of claim 20, wherein the data processing system is a symmetric multiprocessing system.

22. A data processing system comprising:

a set of processors;

a run time system firmware; and

a service processor connected to the set of processors, wherein the run time system firmware and the service processor monitor the set of processors for a selected type of fault, wherein the selected type of fault is a fault occurring often enough to exceed a threshold in which the selected type of fault is an indication of a processor degradation, identify a processor from the set of processors causing the selected type of fault, and dynamically initiate actions to stop the processor in response to detecting the selected type of fault.

23. A data processing system comprising:

a set of processors;

a run time system firmware; and

a service processor connected to the set of processors, wherein the run time system firmware and the service processor monitor the set of processors for a selected type of fault, wherein the selected type of fault is pattern of soft errors trending towards a hard error in which the selected type of fault is an indication of a processor degradation, identify a processor from the set of processors causing the selected type of fault, and dynamically initiate actions to stop the processor in response to detecting the selected type of fault.

24. A data processing system comprising:

a set of processors;

a run time system firmware; and

a service processor connected to the set of processors, wherein the run time system firmware and the service processor monitor the set of processors for a selected type of fault, wherein the selected type of fault is a soft fault in which the selected type of fault is an indication of a processor degradation, identify a processor from the set of processors causing the selected type of fault, and dynamically initiate actions to stop the processor in response to detecting the selected type of fault.

25. The data processing system of claim 24, wherein the processor is removed from the system configuration during a next system boot of the data processing system.

26. A data processing system comprising:

a set of processors;

an operating system;

a run time system firmware; and

a service processor connected to the set of processors, wherein the run time system firmware and the service processor monitor the set of processors for a selected type of fault in which the selected type of fault is an indication of a processor degradation, identify a processor from the set of processors causing the selected type of fault, and dynamically initiate actions to stop the processor in response to detecting the selected type of fault and wherein the run time system firmware initiates actions to stop the processor by sending an error report to the operating system and wherein the operating system migrates processes off of the processor.

27. A data processing system comprising:

a set of processors;

a run time system firmware; and

a service processor connected to the set of processors, wherein the run time system firmware and the service processor monitor the set of processors for a selected type of fault in which the selected type of fault is an indication of a processor degradation, identify a processor from the set of processors causing the selected type of fault, and dynamically initiate actions to stop the processor in response to detecting the selected type of fault, wherein the data processing system includes a system configuration to handle the set of processors and wherein the run time system firmware and the service processor initiates actions to change the system configuration to handle the set of processors minus the processor being stopped.

28. A data processing system for managing a plurality of processors, the data processing system comprising:

monitoring means for monitoring for recoverable errors in a set of processors;

determining means, responsive to detecting a recoverable error for a processor in the set of processors, for determining whether the recoverable error indicates a trend towards an unrecoverable error; and

initiating means, responsive to a determination that the recoverable error indicates a trend towards an unrecoverable error, for initiating actions to stop the processor.

29. The data processing system of claim 28 further comprising:

maintaining means, responsive to detecting a recoverable error for a processor in the set of processors, for maintaining a record of the recoverable error.

30. The data processing system of claim 29, wherein the means of determining comprises:

determining means for determining whether the record indicates a pattern of recoverable errors trending towards a hard error.

31. The data processing system of claim 30, wherein the pattern is a frequency of errors exceeding a threshold within a predefine time period.

32. The data processing system of claim 28, wherein the means for initiating includes:

initiating means for initiating a migration of processes from the processor to other processors in the multiprocessor data processing system.

33. The data processing system of claim 28, wherein the means for initiating includes:

initiating means for initiating a migration of interrupt handlers from the processor to other processors in the multiprocessor data processing system.

34. The data processing system of claim 33, wherein the migration of interrupt handlers are handled by an operating system for the multiprocessor data processing system.

35. The data processing system of claim 28, wherein the means for initiating includes:

stopping means for stopping dispatch tasks on the processor.

36. The data processing system of claim 28, wherein the means for initiating includes:

stopping means for stopping inter-processor interrupts to the processor.

37. The data processing system of claim 28, wherein the means for initiating includes:

storing means for storing data from a cache used by the processor in a memory.

38. The data processing system of claim 28 further comprising:

maintaining means for maintaining a record of the processor, wherein the processors in the record are left out of the system configuration on a system boot process.

39. The data processing system of claim 28, wherein the monitoring means, determining means, and initiating means are located in firmware for the multiprocessor data processing system.

40. A data processing system for managing a plurality of processors, the data processing system comprising:

monitoring means for monitoring the plurality of processors for errors;

adding means, responsive to detecting a recoverable error localized to a processor within the plurality of processors, for adding the recoverable error to a record of errors for the processor;

determining means, responsive to adding the recoverable error to the record, for determining whether the record indicates a trend towards an occurrence of an unrecoverable error; and

identifying means, responsive to determining that the record indicates a trend towards an occurrence of an unrecoverable error, for identifying the processor as a processor that should be removed.

41. The data processing system of claim 40 further comprising:

sending means for sending the identification of the processor to an operating system for the multiprocessor data processing system to initiate actions to migrate processes away from the processors to other processors within the plurality of processors.

42. The data processing system of claim 40 further comprising:

transitioning means for transitioning the multiprocessor data processing system to manage the plurality of processors minus the processor.

43. The data processing system of claim 42 further comprising:

moving means for moving data from a cache used by the processor to a memory in the multiprocessor data processing system.

44. The data processing system of claim 42 further comprising:

removing means for removing the processors from a system configuration for the multiprocessor data processing system during a next system boot of the multiprocessor data processing system.

45. The data processing system of claim 40, wherein the multiprocessor data processing system is a symmetrical multiprocessor system.

46. The data processing system of claim 40, wherein the monitoring means, adding means, determining means, and identifying means are located in a run time system firmware and a service processor in the multiprocessor data processing system.

47. A computer program product in a computer readable medium for managing a plurality of processors, the computer program product comprising:

first instructions for monitoring for recoverable errors in a set of processors;

second instructions, responsive to detecting a recoverable error for a processor in the set of processors, for determining whether the recoverable error indicates a trend towards an unrecoverable error; and

third instructions, responsive to a determination that the recoverable error indicates a trend towards an unrecoverable error, for initiating actions to stop the processor.

48. A computer program product in a computer readable medium for managing a plurality of processors, the computer program product comprising:

first instructions for monitoring the plurality of processors for errors;

second instructions, responsive to detecting a recoverable error localized to a processor within the plurality of processors, for adding the recoverable error to a record of errors for the processor;

third instructions, responsive to adding the recoverable error to the record, for determining whether the record indicates a trend towards an occurrence of an unrecoverable error; and

fourth instructions, responsive to determining that the record indicates a trend towards an occurrence of an unrecoverable error, for identifying the processor as a processor that should be removed.