PURGING OF STORED TIMESERIES DATA

• US 20080183778A1
• Filed: 04/03/2008
• Published: 07/31/2008
• Est. Priority Date: 11/22/2006
• Status: Active Grant

• Current Assignee: Domo, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Neogi, Anindya, Kothari, Ravi, Singh, Raghavendra
• Granted Patent: US 8,001,093 B2
• Time in Patent Office: Days
• US Class Current: 707/206
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures

Abstract

There is disclosed methods, systems and computer program products for purging stored data in a repository. Users attach relative importance to all data samples across all timeseries in a repository. The importance attached to a data sample is the ‘utility value’ of the data sample. An algorithm uses the utility of data samples and allocates the storage space of the repository in such a way that the total loss of information due to purging is minimized while preserving samples with a high utility value.

Claims

1. A method for purging timeseries data samples stored in a repository comprising:

calculating a utility value for each said data sample;

determining information content of each said data sample; and

purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized.

2. The method of claim 1, all the limitations of which are incorporated herein by reference, wherein said purging also ensures that a maximum capacity of said repository is not exceeded.

3. The method of claim 1, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on relationships between said stored data.

4. The method of claim 3, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on a dependency model being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes and edges representing dependencies between said data samples, and each said data sample being associated with a node.

5. The method of claim 1, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on regions of said data samples of interest.

6. The method of claim 5, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with regions of interest.

7. The method of claim 1, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on the age of said data samples.

8. The method of claim 7, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with age.

9. The method of claim 8, all the limitations of which are incorporated herein by reference, wherein age is determined by a polynomic function.

10. The method of claim 1, all the limitations of which are incorporated herein by reference, wherein determination of said information content of said stored data samples is performed on the basis of using least one:

of a probability distribution function, a mean square error, and a Kullback Liebler distance applied to said data samples.

11. A method for purging stored timeseries data comprising:

specifying meta-data purging policy rules and meta-data models of said stored data;

applying utility values to said stored data; and

purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized, and wherein said information content is determined based on said purging policy rules.

12. A method for purging timeseries data samples stored in a repository comprising:

calculating a utility value for each said data samples;

determining information content of each said data sample; and

purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized, and such that a maximum capacity of said repository is not exceeded.

13. The method of claim 12, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on relationships between said stored data.

14. The method of claim 13, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on a dependency model being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes and edges representing dependencies between said data samples, and each said data sample being associated with a node.

15. The method of claim 12, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on regions of said timeseries data samples of interest.

16. The method of claim 15, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with regions of interest.

17. The method of claim 12, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on the age of said data samples.

18. The method of claim 17, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with age.

19. The method of claim 12, all the limitations of which are incorporated herein by reference, wherein determination of said information content of said stored data samples is performed on the basis of using least one:

of a probability distribution function, a mean square error, and a Kullback Liebler distance applied to said data samples.

20. A system comprising:

a repository storing timeseries data samples; and

a processor for calculating a utility value for each said data samples, determining information content of each said data sample, and purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized.

21. The system of claim 20, all the limitations of which are incorporated herein by reference, wherein said purging also ensures that a maximum capacity of said repository is not exceeded.

22. The system of claim 20, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on relationships between said stored data.

23. The system of claim 22, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on a dependency model being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes and edges representing dependencies between said data samples, and each said data samples being associated with a node.

24. The system of claim 20, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on regions of said data samples of interest.

25. The system of claim 24, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with regions of interest.

26. The system of claim 20, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on the age of said data samples.

27. The system of claim 26, all the limitations of which are incorporated herein by reference, wherein said utility value calculation is based on a dependency model of said data samples being represented as a hierarchical graph having (i) nodes, and (ii) edges interconnecting nodes, said nodes being associated with age.

28. The system of claim 20, all the limitations of which are incorporated herein by reference, wherein determination of said information content of said stored data samples is performed on the basis of using least one:

of a probability distribution function, a mean square error, and a Kullback Liebler distance applied to said data samples.

29. A system for purging stored data comprising:

a repository storing timeseries data samples;

a memory specifying meta-data purging policy rules and meta-data models of said stored data samples; and

a processor applying utility values to said stored data, and purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized, and wherein said information content is determined based on said purging policy rules.

30. A system for purging stored data, comprising:

a repository storing timeseries data samples; and

a processor calculating a utility value for each said data samples, determining information content of each said data sample, and purging said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized, and such that a maximum capacity of said repository is not exceeded.

31. A computer program product comprising a computer useable medium having a computer readable program, wherein the computer readable program when executed on a computer causes the computer to:

calculate a utility value for each said data samples;

determine information content of each said data sample; and

purge said stored data samples from said repository such that data samples having high utility value are retained and loss of information content of retained data samples is minimized and such that a maximum capacity of said repository is not exceeded.