Enterprise management system and method which includes summarization having a plurality of levels of varying granularity

• US 6,513,065 B1
• Filed: 03/04/1999
• Issued: 01/28/2003
• Est. Priority Date: 03/04/1999
• Status: Expired due to Term

• Current Assignee: BMC Software Incorporated (KKR & Co., Inc.)
• Original Assignee: BMC Software Incorporated (KKR & Co., Inc.)
• Inventors: Rocco, Joseph, Agrawal, Subhash, Hafez, Amr
• Primary Examiner(s): Sheikh, Ayaz
• Assistant Examiner(s): Duong, Oanh
• Application Number:
  US09/262,194
• Time in Patent Office: 1,426 Days
• Field of Search: 709/202, 709/224, 709/223, 709/203, 709/231, 709/331, 709/204, 707/101, 707/7, 707/8, 707/200, 707/203, 707/500.1, 370/252, 370/253, 345/134, 345/328
• US Class Current: 709/224
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/022 by sampling
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps

Abstract

A system and method for summarizing metric data with a plurality of levels of varying granularity. The system preferably comprises a distributed computing environment, i.e., an enterprise, which comprises a plurality of interconnected computer systems. At least one of the computer systems is an agent computer system which collects raw data relating to one or more metrics, i.e., measurements of system resources on the agent computer system. A Universal Data Repository (UDR) receives raw metric data from one or more agents. The UDR summarizes the raw data into a more compact yet meaningful form. The UDR can summarize both raw data and data that has previously been summarized one or more times, thus creating a plurality of levels of summarization. With each successive summarization, metric data become more compact, yet the data retain information and meaning. Each level of summarization is coarser in granularity and typically older than the previous level: the metric data representing a given period of time become more summarized and take up less space. So that the record of a particular process is never totally lost, process state changes are preserved throughout. The UDR preferably stores each level of summarization in a different file. When a file fills up to its configured maximum size, the oldest metric data from that file are summarized and pushed into the next coarsest file. When the coarsest file fills up, the oldest metric data from the coarsest file are deleted.

Claims

1. A method for storing data regarding one or more computer systems in an enterprise, the method comprising:

receiving a plurality of raw data points, wherein the raw data points comprise measurements of one or more system resources of the one or more computer systems over a period of time;

summarizing the plurality of raw data points, wherein the summarizing comprises creating one or more summarized data structures, wherein each summarized data structure corresponds to at least two of the raw data points;

summarizing a plurality of the summarized data structures, wherein the summarizing the plurality of summarized data structures;

(a) creates one or more coarser summarized data structures, wherein each coarser summarized data structure corresponds to at least two previously summarized data structures;

(b) occurs a plurality of times in sequence as new raw data points are received to create further coarser summarized data structures, wherein a degree of coarseness of the summarized data structures increases with an age of the raw data points summarized in the summarized data structures.

2. The method of claim 1, wherein the summarizing a plurality of the summarized data structures occurs a plurality of times in sequence to create further coarser summarized data structures, wherein a degree of coarseness of the summarized data structures increases with an age of the raw data points summarized in the summarized data structures.

3. The method of claim 1,

wherein the receiving a plurality of raw data points includes receiving a plurality of sets of data points over a period of time, wherein the plurality of sets includes a youngest set of the raw data points and an oldest set of the raw data points;

wherein the summarizing the plurality of raw data points further comprises summarizing the oldest set of the raw data points.

4. The method of claim 1,

wherein said receiving a plurality of raw data points, said summarizing the plurality of raw data points, and said summarizing a plurality of the summarized data structures are performed respectively for a plurality of sets of raw data points,

wherein each summarized data structure corresponds to at least two sets of raw data points;

wherein a degree of coarseness of the summarized data structures increases with an age of the raw data points summarized in the summarized data structures.

5. The method of claim 1, further comprising storing the raw data points in a raw file, wherein the raw data points are stored sequentially in order of collection.

6. The method of claim 5, wherein the summarizing the plurality of raw data points further comprises summarizing an oldest subset of the raw data points stored in the raw file when the raw file reaches a maximum raw file size.

7. The method of claim 6, further comprising deleting from the raw file the oldest subset of the raw data points.

8. The method of claim 1, further comprising storing the summarized data structures in a summarized file, wherein the summarized data structures are stored sequentially in order of collection of corresponding raw data points.

9. The method of claim 8, wherein the summarizing a plurality of summarized data structures firther comprises summarizing an oldest subset of the summarized data structures when the summarized file reaches a maxinum summarized file size.

10. The method of claim 9, further comprising deleting from the summarized file the oldest subset of the summarized data structures.

11. The method of claim 1,

wherein the raw data points further comprise measurements of one or more processes of the one or more computer systems over the period of time;

wherein the summarizing the plurality of raw data points further comprises determining one or more state changes from the raw data points, wherein the state changes comprise a starting time for each process begun within the period of time and an ending time for each process ended within the period of time;

wherein the summarized data structures include the state changes;

wherein the summarizing a plurality of summarized data structures further comprises determining one or more state changes from the summarized data structures, wherein the state changes comprise a starting time for each process begun within the period of time and an ending time for each process ended within the period of time;

wherein the coarser summarized data structures include the state changes.

12. The method of claim 1, wherein the receiving a plurality of raw data points, the summarizing the plurality of raw data points, and the summarizing a plurality of summarized data structures occur on a single computer system of the one or more computer systems.

13. The method of claim 1, wherein the receiving a plurality of raw data points and the summarizing the plurality of raw data points occur on two or more different computer systems.

14. The method of claim 1, wherein the receiving a plurality of raw data points and the summarizing a plurality of summarized data structures occur on two or more different computer systems.

15. A method for storing data regarding one or more computer systems in an enterprise, the method comprising:

receiving a plurality of sets of raw data points over a period of time, wherein the sets of raw data points comprise measurements of one or more system resources of the one or more computer systems, wherein each set of raw data points has an age, wherein the sets of raw data points are ordered by age, wherein, at any point in time the plurality of sets of raw data points includes a current youngest set of raw data points and a current oldest set of raw data points;

summarizing the raw data points in the current oldest set of raw data points, wherein the summarizing the raw data. points creates one or more summarized data structures, wherein each summarized data structure corresponds to at least two data points of the current oldest set of raw data points;

wherein the summarizing the raw data points occurs a plurality of times for respective sets of raw data points to create a plurality of sets of summarized data structures, wherein each set of summarized data structures has an age, wherein the sets of summarized data structures are ordered by age, wherein, at any point in time, the plurality of sets of summarized data structures includes a current youngest set of summarized data structures and a current oldest set of summarized data structures;

summarizing the summarized data structures in the current oldest set ef summarized data structures, wherein the summarizing the summarized data structures;

(a) creates one or more coarser summarized data structures, wherein each coarser summarized data structure corresponds to at least two summarized data structures of the current oldest set of summarized data structures;

(b) occurs a plurality of times for respective sets of summarized data structures to create a plurality of sets of further coarser summarized data structures, wherein a degree of coarseness of the summarized data structures increases with an age of the raw data points summarized in the summarized data structures.

16. A system for storing data regarding one or more computer systems in an enterprise, the system comprising:

a CPU;

a system memory coupled to the CPU, wherein the system memory stores one or more computer programs executable by the CPU;

wherein the computer programs are executable to;

receive a plurality of raw data points, wherein said raw data points comprise measurements of one or more system resources of the one or more computer systems over a period of time;

summarize the plurality of raw data points, to create one or more summarized data structures, wherein each summarized data structure corresponds to at least two of the raw data points;

summarize a plurality of the summarized data structures, to create one or more coarser summarized data structures, wherein each coarser summarized data structure corresponds to at least two of the previously summarized data structures;

summarize the plurality of the summarized data structures a plurality of times in sequence as new raw data points are received, to create further coarser summarized data structures, wherein a degree of coarseness of the summarized data structure increases with an age of the raw data points summarized in the summarized data structures.

17. The system of claim 16, wherein the computer programs are executable to summarize the plurality of the summarized data structures a plurality of times in sequence to create further coarser summarized data structures, wherein a degree of coarseness of the summarized data structure increases with an age of the raw data points summarized in the summarized data structures.

18. The system of claim 16,

wherein in receiving the plurality of raw data points, the computer programs are executable to receive a plurality of sets of data points over a period of time, wherein the plurality of sets includes a youngest set of the raw data points and an oldest set of the raw data points;

wherein in summarizing the plurality of raw data points, the computer programs are executable to summarize the oldest set of the raw data points.

19. The system of claim 16, wherein the computer programs are executable to receive the plurality of raw data points, summarize the plurality of raw data points, and summarize the plurality of the summarized data structures respectively for a plurality of sets of raw data points, wherein each summarized data structure corresponds to at least two sets of raw data points, wherein a degree of coarseness of the summarized data structures increases with an age of the raw data points summarized in the summarized data structures.

20. The system of claim 16, furher. comprising a nonvolatile memory.

21. The system of claim 20, wherein the nonvolatile memory stores the raw data points in a raw file, wherein the raw data points are stored sequentially in order of collection.

22. The system of claim 21, wherein in summarizing the plurality of raw data points, the computer programs are executable to summarize an oldest subset of the raw data points when the raw file reaches a maximum raw file size.

23. The system of claim 22, wherein the computer programs are executable to delete from the raw file in the nonvolatile memory the oldest subset of the raw data points.

24. The system of claim 20, wherein the nonvolatile memory stores the summarized data structures in a summarized file, wherein the summarized data structures are stored sequentially in order of collection of corresponding raw data points.

25. The system of claim 24, wherein in summarizing the plurality of summarized data structures, the computer programs are executable to summarize an oldest subset of the summarized data structures when the summarized file reaches a maximum summarized file size.

26. The system of claim 25, wherein the computer programs are executable to delete from the summarized file in the nonvolatile memory the oldest subset of the summarized data structures.

27. The system of claim 16,

wherein the raw data points further comprise measurements of one or more processes over the period of time;

wherein the computer programs are executable to determine one or more state changes from the raw data points, wherein the state changes comprise a starting time for each process begun within the period of time and an ending time for each process ended within the period of time;

wherein the summarized data structures include the state changes;

wherein the computer programs are executable to determine one or more state changes from the summarized data structures, wherein the state changes comprise a starting time for each process begun within the period of time and an ending time for each process ended within the period of time;

wherein the coarser summarized data structures include the state changes.

28. A system for storing data regarding one or more computer systems in an enterprise, the system comprising:

a network comprising a plurality of interconnected computer systems, wherein each of the plurality of interconnected computer systems includes one or more system resources;

wherein at least one computer system of the plurality of interconnected computer systems is operable to store a plurality of data points received from the plurality of interconnected computer systems, wherein the plurality of data points comprises a series of measurements of one or more system resources of the plurality of interconnected computer systems over a period of time;

wherein the at least one computer system comprises;

a CPU;

a system memory coupled to the CPU, wherein the system memory stores one or more computer programs executable by the CPU;

wherein the computer programs are executable to;

receive a plurality of raw data points, wherein said raw data points comprise measurements of one or more system resources of the one or more computer systems over a period of time;

summarize the plurality of raw data points, to create one or more summarized data structures, wherein each summarized data structure corresponds to at least two of the raw data points;

summarize a plurality of the summarized data structures, to create one or more coarser summarized data structures, wherein each coarser summarized data structure corresponds to at least two of the previously summarized data structures;

summarize the plurality of the summarized data structures a plurality of times in sequence as new raw data points are received, to create further coarser summarized data structures, wherein a degree of coarseness of the summarized data structure increases with an age of the raw data points summarized in the summarized data structures.

29. The system of claim 28, wherein the computer programs of the at least one computer system are further executable to summarize the plurality of summarized data structures a plurality of times in sequence to create further coarser summarized data structures, wherein a degree of coarseness of the summarized data structure increases with an age of the raw data points summarized in the summarized data structures.

30. The system of claim 28,

wherein in receiving the plurality of raw data points, the computer programs of the at least one computer system are executable to receive a plurality of sets of data points over a period of time, wherein the plurality of sets includes a youngest set of the raw data points and an oldest set of the raw data points;

wherein in summarizing the plurality of raw data points, the computer programs of the at least one computer system are executable to summarize the oldest set of the raw data points.

31. The system of claim 28, wherein the computer programs of the at least one computer system are executable to receive the plurality of raw data points, summarize the plurality of raw data points, and summarize the plurality of the summarized data structures respectively for a plurality of sets of raw data points, wherein each summarized data structure corresponds to at least two sets of raw data points, wherein a degree of coarseness of the summarized data structures increases with an age of the raw data points summarized in the summarized data structures.

32. The system of claim 28, wherein at least one computer system further includes a nonvolatile memory.

33. The system of claim 32, wherein the nonvolatile memory stores the raw data points in a raw file, wherein the raw data points are stored sequentially in order of collection.

34. The system of claim 33, wherein in summarizing the plurality of raw data points, the computer programs are executable to summarize an oldest subset of the raw data points when the raw file reaches a maximum raw file size.

35. The system of claim 34, wherein the computer programs are executable to delete from the raw file in the nonvolatile memory the oldest subset of the raw data points.

36. The system of claim 32, herein the nonvolatile memory stores the summarized data structures in a summarized file, wherein the summarized data structures are stored sequentially in order of collection of corresponding raw data points.

37. The system of claim 36, wherein in summarizing the plurality of summarized data structures, the computer programs are executable to summarize an oldest subset of the summarized data structures when the summarized file reaches a maximum summarized file size.

38. The system of claim 36, wherein the computer programs are executable to delete from the summarized file in the nonvolatile memory the oldest subset of the summarized data structures.

39. The system of claim 28,

wherein the raw data points further comprise measurements of one or more processes over the period of time;

wherein the computer programs are executable to determine one or more state changes from the raw data points, wherein the state changes comprise a starting time for each process begun within the period of time and an ending time for each process ended within the period of time;

wherein the summarized data structures include the state changes;

wherein the computer programs are executable to determine one or more state changes from the summarized data structures, wherein the state changes comprise a starting time for each process begun within the period of time and an ending time for each process ended within the period of time;

wherein the coarser summarized data structures include the state changes.

40. A memory medium which stores program instructions for storing data regarding one or more computer systems in an enterprise, wherein the program instructions are executable to implement:

summarizing a plurality of raw data points, wherein the raw data points comprise measurements of one or more system resources of the one or more computer systems over a period of time, wherein the summarizing comprises creating one or more summarized data structures, wherein each summarized data structure corresponds to at least two of the raw data points;

summarizing a plurality of the summarized data structures, wherein the summarizing the plurality of the summarized data structures;

(a) creates one or more coarser summarized data structures, wherein each coarser summarized data structure corresponds to at least two previously summarized data structures.

(b) occurs a plurality of times in sequence as new raw data points are received to create further coarser summarized data structures, wherein a degree of coarseness of the summarized data structures increases with an age of the raw data points summarized in the summarized data structures.

41. The memory medium of claim 40, wherein the summarizing a plurality of the summarized data structures occurs a plurality of times in sequence to create further coarser summarized data structures, wherein a degree of coarseness of the summarized data structures increases with an age of the raw data points summarized in the summarized data structures.

42. The memory medium of claim 40,

wherein a plurality of sets of raw data points are received over a period of time, and wherein the plurality of sets of raw data points includes a youngest set of the raw data points and an oldest set of the raw data points;

wherein the summarizing a plurality of raw data points further comprises summarizing the oldest set of the raw data points.

43. The memory medium of claim 40,

wherein the summarizing a plurality of raw data points and the summarizing a plurality of the summarized data structures are performed respectively for a plurality of sets of raw data points,

wherein each summarized data structure corresponds to at least two sets of raw data points;

wherein a degree of coarseness of the summarized data structures increases with an age of the raw data points summarized in the summarized data structures.

44. The memory medium of claim 40, wherein the program instructions further implement storing the raw data points in a raw file, wherein the raw data points are stored sequentially in order of collection.

45. The memory medium of claim 44, wherein the summarizing a plurality of raw data points further comprises summarizing an oldest subset of the raw data points stored in the raw file when the raw file reaches a maximum raw file size.

46. The memory medium of claim 45, wherein the program instructions further implement deleting from the raw file the oldest subset of the raw data points.

47. The memory medium of claim 40, wherein the program instructions further implement storing the summarized data structures in a summarized file, wherein the summarized data structures are stored sequentially in order of collection of corresponding raw data points.

48. The memory medium of claim 47, wherein the summarizing a plurality of summarized data structures further comprises summarizing an oldest subset of the summarized data structures when the summarized file reaches a maximum summarized file size.

49. The memory medium of claim 48, wherein the program instructions further implement deleting from the summarized file the oldest subset of the summarized data structures.

50. The memory medium of claim 40,

wherein the raw data points further comprise measurements of one or more processes of the one or more computer systems over the period of time;

wherein the summarizing a plurality of raw data points further comprises determining one or more state changes from the raw data points, wherein the state changes comprise a starting time for each process begun within the period of time and an ending time for each process ended within the period of time;

wherein the summarized data structures include the state changes;

wherein the summarizing a plurality of summarized data structures further comprises determining one or more state changes from the summarized data structures, wherein the state changes comprise a starting time for each process begun within the period of time and an ending time for each process ended within the period of time;

wherein the coarser summarized data structures include the state changes.

51. The memory medium of claim 40, wherein the summarizing a plurality of raw data points and the summarizing a plurality of summarized data structures occur on a single computer system of the one or more computer systems.