Fault-tolerant monitoring apparatus, method and system

• US 10,069,698 B2
• Filed: 04/16/2014
• Issued: 09/04/2018
• Est. Priority Date: 07/23/2013
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Li, Michael
• Primary Examiner(s): Winder, Patrice L
• Assistant Examiner(s): Chouat, Abderrahmen
• Application Number:
  US14/254,313
• Publication Number:
  US 20150032877A1
• Time in Patent Office: 1,602 Days
• Field of Search: 709224, 709226, 709219, 709203, 709223, 709204, 709217, 709221, 709202, 709213, 709220, 709201, 709212, 709245, 709238, 726 1, 726 3, 707737, 707741, 707634, 707610, 707769, 707722, 370329, 370235, 370254
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 11/3096 wherein the means or processing minimize the use of computing system or of computing system component resources, e.g. non-intrusive monitoring which minimizes the probe effect: sniffing, intercepting, indirectly deriving the monitored data from other directly available data
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters

Abstract

A fault-tolerant monitoring apparatus is arranged to monitor physical performance properties of a plurality of networked computing elements, each element including a processing unit and individual memory. The monitoring apparatus comprises a plurality of measurer apparatuses, each arranged to measure the physical performance properties of a single computing element, the physical performance properties being stored as local information in the individual memory of the computing element in which the measurement is made; and one or more collector apparatuses arranged to control collection of remote information representing physical performance properties from individual memory in a plurality of the computing elements; and storage of the remote physical performance information as replicate information in the individual memory of another computing element; wherein the remote physical performance information is collected using third party access.

Claims

1. A fault-tolerant monitoring apparatus arranged to monitor physical performance properties of a plurality of networked computing elements, each computing element comprising a processing unit and individual memory to store local information from the computing element itself and remote information from other computing elements, the monitoring apparatus comprising

a plurality of measurer apparatuses, each arranged to measure the physical performance properties of a single computing element among the plurality of computing elements, the physical performance properties being stored as local information in the individual memory of the single computing element in which the measurement is made;

and

at least one collector apparatus,

each measurer apparatus, among the plurality of measurer apparatuses, being directly linked to a single collector apparatus among the at least one collector apparatus without involvement of a network interface controller to signal the collector apparatus with local information,

each collector apparatus arranged to control,

collection of remote information representing the physical performance properties from the individual memory in a plurality of the computing elements, and

storage of the remote information as replicate information in the individual memory of another computing element;

wherein the remote information is collected by a computing element other than the computing element from which the remote information is collected using third party access, and the remote information is stored as replicate information in the individual memory of the computing element which collected the remote information, or another computing element other than a computing element from which the remote information is collected using the third party access;

wherein the physical performance properties are in form of electrical characteristics including any of voltage, current power or energy use of the computing element or part thereof; and

wherein the third party access is by one of remote direct memory access (RDMA) put and RDMA get.

2. A monitoring apparatus according to claim 1,

wherein the at least one collector apparatus is further arranged to control collection of remote data information representing the data performance properties from the individual memory in the plurality of computing elements by a computing element other than a computing element from which the remote data information is collected and storage of the remote data information as replicate data information in the individual memory of the computing element, which collected the remote data information, or another computing element other than the computing element from which the remote data information is collected;

and

wherein the remote data information is collected using third party access.

3. A monitoring apparatus according to claim 1, wherein at least one measurer apparatus among the plurality of measurer apparatuses includes resistors to estimate one or more of:

voltage, current, power, and energy supplied to one or more components of the single computing element.

4. A monitoring apparatus according to claim 1, wherein the individual memory is volatile memory, comprising pinned RAM, RAM and a cache memory.

5. A monitoring apparatus according to claim 1, wherein the monitoring apparatus is controlled to switch on any one of the at least one collector apparatus and at least one measurer apparatus among the plurality of measurer apparatuses, in accordance with the computing elements currently used in an application.

6. A monitoring apparatus according to claim 1, wherein the at least one collector apparatus is more than one collector apparatus and wherein the monitoring apparatus is controlled to activate collection by a subset of the collector apparatuses, the replicate information being stored in a subset of the computing elements.

7. A monitoring apparatus according to claim 1, wherein the plurality of networked computing elements forms a single computer system, the computing elements acting as nodes, each node comprising the processing unit in a form of at least one CPU, the individual memory as RAM memory and the network interface controller to link to the network.

8. A monitoring apparatus according to claim 7, wherein the monitoring apparatus comprises the plurality of measurer apparatuses for each node and the at least one collector apparatus is more than one collector apparatus, each collector apparatus shared between a group of nodes, and arranged to collect the remote information within the group of nodes and from nodes of other groups.

9. A monitoring apparatus according to claim 8,

wherein each collector apparatus shared between the group of nodes is linked to each node of the group of nodes via the network interface controller, and

wherein each measurer apparatus monitors one or more of the network interface controller, the individual memory and the CPU in a corresponding node and is directly linked to a single collector apparatus.

10. A monitoring apparatus according to claim 9, wherein the monitoring apparatus is controlled to activate measurement by a subset of the measurer apparatuses, and wherein the measurer apparatuses activated are directly linked to activated collector apparatuses.

11. A monitoring apparatus according to claim 9, wherein the at least one collector apparatus includes a controller and storage and the controller is operable to update the storage with data performance information from the measurer apparatuses with which the at least one collector apparatus is directly linked and to write information from the storage to the individual memory of the computing element to which the remote information is stored as location information using third party access.

12. A fault-tolerant monitoring method for monitoring physical performance properties of a plurality of networked computing elements, each computing element including a processing unit and an individual memory to store local information from the computing element itself and remote information from other computing elements, the monitoring method comprising

measuring the physical performance properties of corresponding computing elements using measurer apparatuses and storing local information representing the physical performance properties in the individual memory of the corresponding computing elements;

and

by at least one collector apparatus,

collecting remote information representing the physical performance properties from the individual memory in a particular computing element among the computing elements, and

storing the remote information as replicate information in the individual memory of another computing element;

wherein each measurer apparatus, among the measurer apparatuses, is directly linked to a single collector apparatus among the at least one collector apparatus without involvement of a network interface controller to signal the collector apparatus with local information;

wherein the remote information is collected by a computing element other than the computing element from which the remote information is collected using third party access, and the remote information is stored as replicate information in the individual memory of the computing element which collected the remote information, or another computing element other than the computing element from which the remote information is collected, using the third party access;

wherein the physical performance properties are in form of electrical characteristics including any of voltage, current, power or energy use of the computing element or part thereof; and

wherein the third party access is by one of remote direct memory access (RDMA) put and RDMA get.

13. A method according to claim 12, wherein the particular computing element either fails, or is deselected during execution of an application;

and

the application either restarts from a checkpoint, or continues, omitting the particular computing element,

wherein the physical performance data for the particular computing element before failure is provided by the other computing element in which the remote information representing the physical performance data was stored as the replicate information.

14. A non-transitory computer-readable medium tangibly embodying a computer program, which when loaded onto a distributed memory computer system with a plurality of networked computing elements, each computing element including a processing unit and an individual memory to store local information from the computing element itself and remote information from other computing elements, configures the distributed memory computer system to:

measure physical performance properties of corresponding computing elements using measurer apparatuses and store local information representing the physical performance properties in individual memory of the corresponding computing elements; and

using at least one collector apparatus,

collect remote information representing the physical performance properties from the individual memory in a particular computing element among the computing elements, and

store the remote information as replicate information in the individual memory of another computing element;

wherein each measurer apparatus, among the measurer apparatuses, is directly linked to a single collector apparatus among the at least one collector apparatus without involvement of a network interface controller to signal the collector apparatus with local Information;

wherein the remote information is collected by a computing element other than the computing element from which the remote information is collected using third party access, and the remote information is stored as replicate information in the individual memory of the computing element which collected the remote information, or another computing element other than the computing element from which the remote information is collected, using the third party access;

wherein the physical performance properties are in form of electrical characteristics including any of voltage, current, power or energy use of the computing element or part thereof; and

wherein the third party access is by one of remote direct memory access (ROMA) put and RDMA get.

15. A computer system comprising a plurality of networked computing elements, each computing element including a processing unit and individual memory to store local information from the computing element itself and remote information from other computing elements, the computer system including a fault-tolerant monitoring apparatus arranged to monitor physical performance properties of the networked computing elements, the monitoring apparatus comprising:

a plurality of measurer apparatuses each arranged to measure the physical performance properties of a single computing element among the plurality of computing elements, for storage as local information in the individual memory of the particular single computing element; and

at least one collector apparatus,

each measurer apparatus, among the plurality of measurer apparatuses, being directly linked to a single collector apparatus among the at least one collector apparatus without involvement of a network interface controller to signal the collector apparatus with local information;

each collector apparatus arranged to control,

collection of remote information representing the physical performance properties from individual memory in a plurality of the computing elements, and

storage of the remote information as replicate information in the individual memory of another computing element;

wherein the remote information is collected by a computing element other than the computing element from which the remote information is collected using third party access, and the remote information is stored as replicate information in the individual memory of the computing element which collected the remote Information, or another computing element other than the computing element from which the remote information is collected, using the third party access;

wherein the physical performance properties are in form of electrical characteristics including any of voltage, current, Dower or energy use of the computing element or part thereof; and

wherein the third party access is by one of remote direct memory access (RDMA) put and RDMA get.