Method and system for information gathering and aggregation in dynamic distributed environments

• US 20060120384A1
• Filed: 12/08/2004
• Published: 06/08/2006
• Est. Priority Date: 12/08/2004
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Zhou, Nianjun, Boutboul, Irwin, Meliksetian, Dikran S., Prost, Jean-Pierre
• Application Number:
  US11/007,044
• Publication Number:
  US 20060120384A1
• Time in Patent Office: Days
• US Class Current: 370/400
• CPC Class Codes: H04L 45/00 Routing or path finding of packets in data switching networks routing or path finding in wireless networks H04W40/00

Abstract

A method for information gathering and aggregation in a dynamic distributed environment includes obtaining topology information identifying a plurality of topology nodes of the topology and communication paths of the plurality of topology nodes. An information services policy is obtained, and information gathering directives are determined for information gathering nodes included in the plurality of topology nodes and sent thereto, based on the obtained topology information and the obtained information services policy. Information aggregating directives are also determined for information aggregating nodes included in the plurality of topology nodes and sent thereto, based on the obtained topology information and the obtained information services policy.

Claims

1. A method for information gathering and aggregation in a dynamic distributed environment comprising a topology of topology nodes, and information aggregation comprising any one of analyzing or condensing gathered information, the method comprising the steps of:

obtaining topology information, the topology information identifying a plurality of topology nodes of the topology and communication paths of the plurality of topology nodes;

obtaining an information services policy;

determining information gathering directives for information gathering nodes included in the plurality of topology nodes, based on the obtained topology information and the obtained information services policy;

determining information aggregating directives for information aggregating nodes included in the plurality of topology nodes, based on the obtained topology information and the obtained information services policy;

sending the information gathering directives to the information gathering nodes of the plurality of topology nodes; and

sending the information aggregating directives to the information aggregating nodes of the plurality of topology nodes.

4. The method of claim 1, wherein the topology nodes further comprise:

one or more leaf nodes having only incoming edges thereto, at least one of the one or more leaf nodes configured to collect information about itself;

one or more prime nodes having both incoming and outgoing edges, at least of the one or more prime nodes configured to aggregate the information received from other topology nodes to which the configured prime node subscribes; and

one or more root nodes having only outgoing edges, at least one of the one or more root nodes configured to aggregate information received from other topology nodes to which the configured root node subscribes.

5. The method of claim 1, wherein aggregation comprises one or more of any one of:

assigning ranges of values and categorizing a given node based on the range into which the value reported by the given node falls;

rounding the values reported by the given node into coarser ranges;

eliminating information that does not change from previously reported values thereof;

or

categorizing nodes into categories rather than reporting the corresponding information values.

6. The method of claim 1, wherein data gathering comprises one or more of any one of:

subscribing for notification of information from one or more nodes when the value of the information changes by a policy-specified amount;

subscribing for periodic notification of the information from the one or more nodes;

or

querying the one or more nodes for specific up to date information.

7. The method of claim 4, wherein the information collected includes any one of CPU usage or memory.

8. The method of claim 1, wherein the information services policy comprises one or more of any one of:

maintaining a balance between information gathering cost and information querying cost, by making the aggregation ranges coarser or finer;

or

ensuring that prime nodes spend less than a predefined proportion of their resources for the tasks of information aggregation by creating new prime nodes and distributing the information gathering load to the created new prime nodes.

9. The method of claim 1, further comprising:

receiving a communication from an entity, the communication including a topology event notification indicating an event affecting the topology;

determining with an automated topology formation application that the topology event notification affects a topology portion of the topology; and

based on the topology event notification, modifying the affected topology portion of the topology according to a predefined topology policy definition.

10. The method of claim 9, wherein the topology event notification is initiated by an aggregating node detecting a need for a topology change.

11. The method of claim 4, wherein a first of the prime nodes subscribes directly to at least one of the one or more leaf nodes, and aggregates information therefrom at a first level of granularity.

12. The method of claim 11, wherein a second of the prime nodes subscribes directly to at least another of the one or more prime nodes, and aggregates information therefrom at a second desired level of granularity.

13. The method of claim 12, wherein the second level of granularity is coarser than the first level of granularity.

14. The method of claim 4, wherein the master node is configured to notify the root node of any change in the topology.

15. The method of claim 4, wherein information aggregated by at least one of the one or more prime nodes is updated based upon any one of a predefined frequency or a predefined event.

16. The method of claim 4, wherein said root node provides a meta indexing service component of the information services policy.

17. The method of claim 16, wherein the meta indexing service component provides information of the roles of the prime nodes and provides a registry service for the prime nodes.

18. The method of claim 1, further comprising the steps of:

configuring a master node in the topology, the master node further comprising an automated topology formation application having a predefined topology policy definition and a representation of the topology, wherein the topology comprises nodes and intercommunication paths between the nodes, the nodes further comprising;

one or more leaf nodes having only incoming edges thereto, one or more of the one or more leaf node configured to collect information about itself;

one or more prime nodes having both incoming and outgoing edges, one or more of the one or more prime nodes configured to information received from other nodes to which each said prime node subscribes, based on a predefined information services policy; and

one or more root prime nodes having only outgoing edges, one or more of the one or more root prime nodes configured to aggregate and index information received from other nodes to which each said root prime node subscribes;

the master node further comprising an automated topology formation application having a predefined topology policy definition and a representation of the active topology;

transmitting collected information from a configured leaf node to a subscribing prime node, the collected information collected according to a collecting directive, the transmitting performed according to a predetermined schedule;

receiving the transmitted collected information at a first configured prime node and aggregating the received collected information with collected information received from one or more other configured leaf nodes;

when the first configured prime node is subscribed to by a second configured prime node, transmitting the aggregated information to the second configured prime node according to a predetermined schedule;

when the first configured prime node is subscribed to by a second configured prime node, aggregating at the second configured prime node the aggregated information with information received from other nodes subscribed to by the second configured prime node;

when an aggregating step detects a predefined topology affecting event, transmitting a topology event notification to the master node, the event notification indicating an event affecting the active topology;

determining with the automated topology formation application that the topology event notification affects a topology portion of the active topology; and

based on the topology event notification, modifying the representation of the affected topology portion of the active topology according to the predefined topology policy definition.

2. The method of claim, further comprising periodically updating the topology information.

3. The method of claim 2, wherein the topology information is updated due to any one of a change of nodes of the topology, a change of communication paths of the topology or a detected failure of an aggregating node, the detected failure triggering the selection of a substitute aggregating node.

19. A method for information gathering and aggregation in a dynamic distributed environment comprising an active topology of topology nodes, and information aggregation comprising one or more of analyzing or condensing gathered information, the method comprising the steps of:

configuring a master node in the active topology, the master node comprising an automated topology formation application having a predefined topology policy definition and a representation of the active topology, wherein the active topology comprises nodes and intercommunication paths between the nodes, the nodes further comprising;

one or more leaf nodes having only incoming edges thereto, one or more of the one or more leaf node configured to collect information about itself;

one or more prime nodes having both incoming and outgoing edges, one or more of the one or more prime nodes configured to information received from other nodes to which each said prime node subscribes, based on a predefined information services policy; and

one or more root prime nodes having only outgoing edges, one or more of the one or more root prime nodes configured to aggregate and index information received from other nodes to which each said root prime node subscribes;

transmitting collected information from a configured leaf node to a subscribing prime node, the collected information collected according to a collecting directive, the transmitting performed according to a predetermined schedule;

receiving the transmitted collected information at a first configured prime node and aggregating the received collected information with collected information received from one or more other configured leaf nodes;

when the first configured prime node is subscribed to by a second configured prime node, transmitting the aggregated information to the second configured prime node according to a predetermined schedule;

when the first configured prime node is subscribed to by a second configured prime node, aggregating at the second configured prime node the aggregated information with information received from other nodes subscribed to by the second configured prime node;

when an aggregating step detects a predefined topology affecting event, transmitting a topology event notification to the master node, the event notification indicating an event affecting the active topology;

determining with the automated topology formation application that the topology event notification affects a topology portion of the active topology; and

based on the topology event notification, modifying the representation of the affected topology portion of the active topology according to the predefined topology policy definition.

20. The method of claim 19, wherein the active topology information is modified due to any one of a change of nodes of the topology, a change of intercommunication paths of the topology or a detected failure of an aggregating node the detected failure triggering the selection of a substitute aggregating node.

21. The method of claim 20, further comprising substituting a new aggregating node of the topology upon detection of a failed aggregating node of the topology.

22. The method of claim 19, wherein aggregation comprises one or more of any one of:

assigning ranges of values and categorizing a given node based on the range into which the value reported by the given node falls;

rounding the values reported by the given node into coarser ranges;

eliminating information that does not change from previously reported values thereof;

or

categorizing nodes into categories rather than reporting the corresponding information values.

23. The method of claim 19, wherein data gathering comprises one or more of:

subscribing for notification of information from one or more nodes when the value of the information changes by a policy-specified amount;

subscribing for periodic notification of the information from the one or more nodes;

or

querying the one or more nodes for specific up to date information.

24. The method of claim 23, wherein the information collected includes any one of CPU usage or memory.

25. The method of claim 19, wherein the information services policy comprises one or more of any one of:

maintaining a balance between information gathering cost and information querying cost, by making the aggregation ranges coarser or finer;

or

ensuring that prime nodes spend less than a predefined proportion of their resources for the tasks of information aggregation by creating new prime nodes and distributing the information gathering load to the created new prime nodes.

26. The method of claim 19, wherein a first of the prime nodes subscribes directly to at least one of the one or more leaf nodes, and aggregates information therefrom at a first level of granularity.

27. The method of claim 26, wherein a second of the prime nodes subscribes directly to at least another of the one or more prime nodes, and aggregates information therefrom at a second desired level of granularity.

28. The method of claim 27, wherein the second level of granularity is coarser than the first level of granularity.

29. The method of claim 19, wherein the master node is configured to notify the root node of any change in the topology.

30. The method of claim 19, wherein information aggregated by at least one of the one or more prime nodes is updated based upon any one of:

a predefined frequency or a predefined event.

31. The method of claim 19, wherein said root node provides a meta indexing service component of the information services policy.

32. The method of claim 31, wherein the meta indexing service component provides information of the roles of the prime nodes and provides a registry service for the prime nodes.

33. The method of claim 19, further comprising distributing any one of aggregating directives or collecting directives to any one of one or more configured leaf nodes or one or more configured prime nodes.

34. A computer program product for information gathering and aggregation in a dynamic distributed environment comprising a topology of topology nodes, and information aggregation comprising any one of analyzing or condensing gathered information, comprising:

a storage medium readable by a processing circuit, the storage medium comprising machine readable computer program code including instructions for causing the processing circuit to implement a method, the method further comprising the steps of;

obtaining topology information, the topology information identifying a plurality of topology nodes of the topology and communication paths of the plurality of topology nodes;

obtaining an information services policy;

determining information gathering directives for information gathering nodes included in the plurality of topology nodes, based on the obtained topology information and the obtained information services policy;

determining information aggregating directives for information aggregating nodes included in the plurality of topology nodes, based on the obtained topology information and the obtained information services policy;

sending the information gathering directives to the information gathering nodes of the plurality of topology nodes; and

sending the information aggregating directives to the information aggregating nodes of the plurality of topology nodes.

35. The computer program product of claim 34, wherein the topology nodes further comprise:

one or more leaf nodes having only incoming edges thereto, at least one of the one or more leaf nodes configured to collect information about itself;

one or more prime nodes having both incoming and outgoing edges, at least of the one or more prime nodes configured to aggregate the information received from other topology nodes to which the configured prime node subscribes; and

one or more root nodes having only outgoing edges, at least one of the one or more root nodes configured to aggregate information received from other topology nodes to which the configured root node subscribes.

36. The computer program product of claim 34, wherein:

aggregation comprises one or more of any one of;

assigning ranges of values and categorizing a given node based on the range into which the value reported by the given node falls;

rounding the values reported by the given node into coarser ranges;

eliminating information that does not change from previously reported values thereof;

or

categorizing nodes into categories rather than reporting the corresponding information values; and

data gathering comprises one or more of any one of;

subscribing for notification of information from one or more nodes when the value of the information changes by a policy-specified amount;

subscribing for periodic notification of the information from the one or more nodes;

or

querying the one or more nodes for specific up to date information.

37. The computer program product of claim 34, further comprising:

receiving a communication from an entity, the communication including a topology event notification indicating an event affecting the topology;

determining with an automated topology formation application that the topology event notification affects a topology portion of the topology; and

based on the topology event notification, modifying the affected topology portion of the topology according to a predefined topology policy definition.

38. The computer program product of claim 34, further comprising the steps of:

configuring a master node in the topology, the master node further comprising an automated topology formation application having a predefined topology policy definition and a representation of the topology, wherein the topology comprises nodes and intercommunication paths between the nodes, the nodes further comprising;

one or more leaf nodes having only incoming edges thereto, one or more of the one or more leaf node configured to collect information about itself;

one or more prime nodes having both incoming and outgoing edges, one or more of the one or more prime nodes configured to information received from other nodes to which each said prime node subscribes, based on a predefined information services policy; and

one or more root prime nodes having only outgoing edges, one or more of the one or more root prime nodes configured to aggregate and index information received from other nodes to which each said root prime node subscribes;

the master node further comprising an automated topology formation application having a predefined topology policy definition and a representation of the active topology;

transmitting collected information from a configured leaf node to a subscribing prime node, the collected information collected according to a collecting directive, the transmitting performed according to a predetermined schedule;

receiving the transmitted collected information at a first configured prime node and aggregating the received collected information with collected information received from one or more other configured leaf nodes;

when the first configured prime node is subscribed to by a second configured prime node, transmitting the aggregated information to the second configured prime node according to a predetermined schedule;

when the first configured prime node is subscribed to by a second configured prime node, aggregating at the second configured prime node the aggregated information with information received from other nodes subscribed to by the second configured prime node;

when an aggregating step detects a predefined topology affecting event, transmitting a topology event notification to the master node, the event notification indicating an event affecting the active topology;

determining with the automated topology formation application that the topology event notification affects a topology portion of the active topology; and

based on the topology event notification, modifying the representation of the affected topology portion of the active topology according to the predefined topology policy definition.

47. The system of claim 36, wherein:

aggregation comprises one or more of any one of;

assigning ranges of values and categorizing a given node based on the range into which the value reported by the given node falls;

rounding the values reported by the given node into coarser ranges;

eliminating information that does not change from previously reported values thereof;

or

categorizing nodes into categories rather than reporting the corresponding information values; and

data gathering comprises one or more of any one of;

subscribing for notification of information from one or more nodes when the value of the information changes by a policy-specified amount;

subscribing for periodic notification of the information from the one or more nodes;

or

querying the one or more nodes for specific up to date information.

39. A computer program product for information gathering and aggregation in a dynamic distributed environment comprising a topology of topology nodes, and information aggregation comprising any one of analyzing or condensing gathered information, comprising:

a storage medium readable by a processing circuit, the storage medium comprising machine readable computer program code including instructions for causing the processing circuit to implement a method, the method further comprising the steps of;

configuring a master node in the active topology, the master node comprising an automated topology formation application having a predefined topology policy definition and a representation of the active topology, wherein the active topology comprises nodes and intercommunication paths between the nodes, the nodes further comprising;

one or more leaf nodes having only incoming edges thereto, one or more of the one or more leaf node configured to collect information about itself;

one or more prime nodes having both incoming and outgoing edges, one or more of the one or more prime nodes configured to information received from other nodes to which each said prime node subscribes, based on a predefined information services policy; and

one or more root prime nodes having only outgoing edges, one or more of the one or more root prime nodes configured to aggregate and index information received from other nodes to which each said root prime node subscribes;

transmitting collected information from a configured leaf node to a subscribing prime node, the collected information collected according to a collecting directive, the transmitting performed according to a predetermined schedule;

receiving the transmitted collected information at a first configured prime node and aggregating the received collected information with collected information received from one or more other configured leaf nodes;

when the first configured prime node is subscribed to by a second configured prime node, transmitting the aggregated information to the second configured prime node according to a predetermined schedule;

when the first configured prime node is subscribed to by a second configured prime node, aggregating at the second configured prime node the aggregated information with information received from other nodes subscribed to by the second configured prime node;

when an aggregating step detects a predefined topology affecting event, transmitting a topology event notification to the master node, the event notification indicating an event affecting the active topology;

determining with the automated topology formation application that the topology event notification affects a topology portion of the active topology; and

based on the topology event notification, modifying the representation of the affected topology portion of the active topology according to the predefined topology policy definition.

40. The computer program product of claim 39, wherein:

aggregation comprises one or more of any one of;

assigning ranges of values and categorizing a given node based on the range into which the value reported by the given node falls;

rounding the values reported by the given node into coarser ranges;

eliminating information that does not change from previously reported values thereof;

or

categorizing nodes into categories rather than reporting the corresponding information values; and

data gathering comprises one or more of any one of;

subscribing for notification of information from one or more nodes when the value of the information changes by a policy-specified amount;

subscribing for periodic notification of the information from the one or more nodes;

or

querying the one or more nodes for specific up to date information.

41. A system for information gathering and aggregation in a dynamic distributed environment comprising a topology of topology nodes, and information aggregation comprising any one of analyzing or condensing gathered information, comprising:

a network;

a first computer system in communication with the network, wherein the first computer system includes instructions for implementing a method, the method further comprising;

a storage medium readable by a processing circuit, the storage medium comprising machine readable computer program code including instructions for causing the processing circuit to implement a method, the method further comprising the steps of;

obtaining topology information, the topology information identifying a plurality of topology nodes of the topology and communication paths of the plurality of topology nodes;

obtaining an information services policy;

determining information gathering directives for information gathering nodes included in the plurality of topology nodes, based on the obtained topology information and the obtained information services policy;

determining information aggregating directives for information aggregating nodes included in the plurality of topology nodes, based on the obtained topology information and the obtained information services policy;

sending the information gathering directives to the information gathering nodes of the plurality of topology nodes; and

sending the information aggregating directives to the information aggregating nodes of the plurality of topology nodes.

42. The system of claim 41, wherein the topology nodes further comprise:

one or more leaf nodes having only incoming edges thereto, at least one of the one or more leaf nodes configured to collect information about itself;

one or more prime nodes having both incoming and outgoing edges, at least of the one or more prime nodes configured to aggregate the information received from other topology nodes to which the configured prime node subscribes; and

one or more root nodes having only outgoing edges, at least one of the one or more root nodes configured to aggregate information received from other topology nodes to which the configured root node subscribes.

43. The system of claim 41, wherein:

aggregation comprises one or more of any one of;

assigning ranges of values and categorizing a given node based on the range into which the value reported by the given node falls;

rounding the values reported by the given node into coarser ranges;

eliminating information that does not change from previously reported values thereof;

or

categorizing nodes into categories rather than reporting the corresponding information values; and

data gathering comprises one or more of any one of;

subscribing for notification of information from one or more nodes when the value of the information changes by a policy-specified amount;

subscribing for periodic notification of the information from the one or more nodes;

or

querying the one or more nodes for specific up to date information.

44. The system of claim 41, further comprising:

receiving a communication from an entity, the communication including a topology event notification indicating an event affecting the topology;

determining with an automated topology formation application that the topology event notification affects a topology portion of the topology; and

based on the topology event notification, modifying the affected topology portion of the topology according to a predefined topology policy definition.

45. The system of claim 41, further comprising the steps of:

configuring a master node in the topology, the master node further comprising an automated topology formation application having a predefined topology policy definition and a representation of the topology, wherein the topology comprises nodes and intercommunication paths between the nodes, the nodes further comprising;

one or more leaf nodes having only incoming edges thereto, one or more of the one or more leaf node configured to collect information about itself;

one or more prime nodes having both incoming and outgoing edges, one or more of the one or more prime nodes configured to information received from other nodes to which each said prime node subscribes, based on a predefined information services policy; and

one or more root prime nodes having only outgoing edges, one or more of the one or more root prime nodes configured to aggregate and index information received from other nodes to which each said root prime node subscribes;

the master node further comprising an automated topology formation application having a predefined topology policy definition and a representation of the active topology;

transmitting collected information from a configured leaf node to a subscribing prime node, the collected information collected according to a collecting directive, the transmitting performed according to a predetermined schedule;

receiving the transmitted collected information at a first configured prime node and aggregating the received collected information with collected information received from one or more other configured leaf nodes;

when the first configured prime node is subscribed to by a second configured prime node, transmitting the aggregated information to the second configured prime node according to a predetermined schedule;

when the first configured prime node is subscribed to by a second configured prime node, aggregating at the second configured prime node the aggregated information with information received from other nodes subscribed to by the second configured prime node;

when an aggregating step detects a predefined topology affecting event, transmitting a topology event notification to the master node, the event notification indicating an event affecting the active topology;

determining with the automated topology formation application that the topology event notification affects a topology portion of the active topology; and

based on the topology event notification, modifying the representation of the affected topology portion of the active topology according to the predefined topology policy definition.

46. A system for information gathering and aggregation in a dynamic distributed environment comprising a topology of topology nodes, and information aggregation comprising any one of analyzing or condensing gathered information, comprising:

a network;

a first computer system in communication with the network, wherein the first computer system includes instructions for implementing a method, the method further comprising;

configuring a master node in the active topology, the master node comprising an automated topology formation application having a predefined topology policy definition and a representation of the active topology, wherein the active topology comprises nodes and intercommunication paths between the nodes, the nodes further comprising;

one or more leaf nodes having only incoming edges thereto, one or more of the one or more leaf node configured to collect information about itself;

one or more prime nodes having both incoming and outgoing edges, one or more of the one or more prime nodes configured to information received from other nodes to which each said prime node subscribes, based on a predefined information services policy; and

one or more root prime nodes having only outgoing edges, one or more of the one or more root prime nodes configured to aggregate and index information received from other nodes to which each said root prime node subscribes;

transmitting collected information from a configured leaf node to a subscribing prime node, the collected information collected according to a collecting directive, the transmitting performed according to a predetermined schedule;

receiving the transmitted collected information at a first configured prime node and aggregating the received collected information with collected information received from one or more other configured leaf nodes;

when the first configured prime node is subscribed to by a second configured prime node, transmitting the aggregated information to the second configured prime node according to a predetermined schedule;

when the first configured prime node is subscribed to by a second configured prime node, aggregating at the second configured prime node the aggregated information with information received from other nodes subscribed to by the second configured prime node;

when an aggregating step detects a predefined topology affecting event, transmitting a topology event notification to the master node, the event notification indicating an event affecting the active topology;

determining with the automated topology formation application that the topology event notification affects a topology portion of the active topology; and

based on the topology event notification, modifying the representation of the affected topology portion of the active topology according to the predefined topology policy definition.

48. A service for information gathering and aggregation in a dynamic distributed environment comprising a topology of intercommunicating topology nodes, wherein aggregating comprises any one of analyzing or condensing gathered information, the topology comprising:

one or more leaf nodes having only incoming edges thereto, one or more of the one or more leaf node configured to collect information about itself;

one or more prime nodes having both incoming and outgoing edges, one or more of the one or more prime nodes configured to information received from other nodes to which each said prime node subscribes, based on a predefined information services policy, wherein the topology comprises a hierarchy of one or more prime nodes; and

one or more root prime nodes having only outgoing edges, one or more of the one or more root prime nodes configured to aggregate and index information received from other nodes to which each said root prime node subscribes;

the service comprising;

configuring a master node to manage the topology;

providing any one of information gathering directives and information aggregating directives to the topology nodes, the directives based on an information services policy wherein any one of information gathered or aggregated information is aggregated at aggregating nodes;

in response to an event notification received at the master node, modifying a portion of the topology wherein an event notification is signaled when any one of a node is added to the topology, a node is removed from the topology, a node fails, an aggregator requests a topology change or an application requests a topology change, wherein further the modification of the portion of the topology comprises any one of modifying a topology node intercommunication path, adding a node, removing a node, modifying the information gathering directive of a node, notifying an application of the event notification or modifying the information aggregating directive of a node.