Optimizing a storage system to support short data lifetimes

• US 7,958,093 B2
• Filed: 09/17/2004
• Issued: 06/07/2011
• Est. Priority Date: 09/17/2004
• Status: Expired due to Fees

• Current Assignee: National Security Agency
• Original Assignee: International Business Machines Corporation
• Inventors: Halim, Nagui, Tetzlaff, William Harold, Richards, Elizabeth Suzanne, Palmer, John Davis, Tao, David, Wolf, Joel Leonard, Anderson, Kay Schwendimann, Tracey, John Michael, Douglis, Frederick
• Primary Examiner(s): Yen; Syling
• Application Number:
  US10/944,597
• Publication Number:
  US 20060072400A1
• Time in Patent Office: 2,454 Days
• Field of Search: 707/206, 707/10, 707/610, 707/640, 707/661, 707/662, 707/667, 707/672, 707/673, 707/674, 707/687, 707/758, 707/781, 707/812, 707/822, 707/999.01, 707/999.107, 707/999.202, 707/999.204, 709/203, 709/223, 709/245, 709/204, 709/205, 709/206, 369/47.1, 711/100, 719/311, 719/315, 719/316
• US Class Current: 707/662
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

A system and method for optimizing a storage system to support short data object lifetimes and highly utilized storage space are provided. With the system and method, data objects are clustered based on when they are anticipated to be deleted. When an application stores data, the application provides an indicator of the expected lifetime of the data, which may be a retention value, a relative priority of the data object, or the like. Data objects having similar expected lifetimes are clustered together in common data structures so that clusters of objects may be deleted efficiently in a single operation. Expected lifetimes may be changed by applications automatically. The system automatically determines how to handle these changes in expected lifetime using one or more of copying the data object, reclassifying the container in which the data object is held, and ignoring the change in expected lifetime for a time to investigate further changes in expected lifetime of other data objects.

Claims

1. A method of storing data in a data storage system, comprising:

receiving a plurality of data objects, wherein each data object of the plurality of data objects has an associated expected lifetime identifier that identifies criteria indicative of an amount of time the associated data object is expected to be retained in the data storage system, wherein each expected lifetime identifier comprises a retention value that indicates a relative importance of retaining the associated data object and an associated modification function for modifying the retention value and wherein the retention value is selected from a range of numbers between and including a first real number and a second real number; and

wherein the first real number is a lowest number of the range of numbers indicating the associated data object is not to be retained and the second real number is a highest number of the range of numbers indicating the associated data object is not to be deleted;

grouping a subset of the data objects into a data container in a storage device based on the data objects having a similar expected lifetime identifier; and

writing the subset of the data objects in the data container out to physical storage in response to an occurrence of an event, wherein the method further comprises;

receiving a modification of a retention value of a data object of the subset of the data objects;

determining whether the data object of the subset of the data objects is to be moved to another data container; and

in response to determining that the data object of the subset of the data objects is to be moved to another data container, associating the data object of the subset of the data objects with another data container containing other data objects that have a retention value that is similar to the modified retention value of the data object of the subset of the data objects.

2. The method of claim 1, wherein each data object of the plurality of data objects is annotated, by an application at data object creation time, with the associated expected lifetime identifier.

3. The method of claim 1, wherein the criteria are one of periods of time or relative priorities.

4. The method of claim 2, wherein the associated modification function defines how the retention value changes over time.

5. The method of claim 4, wherein the associated modification function is a decay function that reduces the retention value over time;

and

wherein as the retention value increases, the amount of time to reduce the retention value increases by an order of magnitude.

6. The method of claim 1, further comprising:

determining if an expected lifetime identifier of a data object of the subset of the data objects in the data container has a predetermined relationship with a deletion threshold; and

deleting all of the data objects of the subset of the data objects in the data container, in response to determining that the expected lifetime identifier of the data object of the subset of the data objects in the data container has the predetermined relationship with the deletion threshold.

7. The method of claim 6, wherein the predetermined relationship is that the expected lifetime identifier has a value that is less than or equal to the value of the deletion threshold, and further comprising dynamically updating a value of the deletion threshold based on a current utilization of the storage system, wherein dynamically updating a value of the deletion threshold includes:

determining a current level of usage of the storage system;

increasing the value of the deletion threshold if the current level of usage of the storage system indicates an increase in usage of the storage system; and

decreasing the value of the deletion threshold if the current level of usage of the storage system indicates a decrease in usage of the storage system.

8. The method of claim 4, wherein each of the expected lifetime identifiers and each of the associated modification functions are maintained in a metadata data structure in memory after writing of the subset of the data objects in the data container out to physical storage, and wherein the data container is deleted from memory after writing of the subset of the data objects in the data container out to physical storage.

9. The method of claim 1, wherein the event is one of the data container becoming full, an elapse of a predetermined period of time, and a flush of the data container.

10. The method of claim 1, wherein the event is detecting that an expected lifetime identifier of a data object of the subset of the data objects in the data container has been changed a predetermined number of times or more in a predetermined period of time, and wherein writing the subset of the data objects in the data container out to physical storage in response to the occurrence of an event includes writing the data object of the subset of the data objects in the data container out to a different physical storage than data objects whose expected lifetime identifiers have not changed a predetermined number of times or more in a predetermined period of time.

11. The method of claim 1, wherein determining whether the data object of the subset of the data objects is to be moved to another data container, comprises:

determining whether the data object of the subset of the data objects is to be moved based on applying data container policies for handling modifications to attributes of data objects.

12. The method of claim 1, and further comprising:

in response to determining that the data object of the subset of the data objects is to be moved to another data container, also marking the data object of the subset of the data objects in the data container for deletion.

13. The method of claim 1, and further comprising:

in response to determining that the data object of the subset of the data objects is to be moved to another data container, determining whether to modify the retention value associated with the data container; and

in response to determining that the retention value associated with the data container is to be modified, updating the retention value associated with the data container.

14. The method of claim 13, and further comprising:

in response to determining that the data object of the subset of the data objects is not to be moved to another data container, updating metadata for the data container.

15. A computer program product for storing data in a data storage system, the computer program product comprising:

a computer usable recordable-type media having computer executable instructions stored thereon, the computer executable instructions comprising;

computer executable instructions for receiving a plurality of data objects, wherein each data object of the plurality of data objects has an associated expected lifetime identifier that identifies criteria indicative of an amount of time the associated data object is expected to be retained in the data storage system, wherein each expected lifetime identifier comprises a retention value that indicates a relative importance of retaining the associated data object and an associated modification function for modifying the retention value and wherein the retention value is selected from a range of numbers between and including a first real number and a second real number; and

wherein the first real number is a lowest number of the range of numbers indicating the associated data object is not to be retained and the second real number is a highest number of the range of numbers indicating the associated data object is not to be deleted;

computer executable instructions for grouping a subset of the data objects into a data container in a storage device based on the data objects having a similar expected lifetime identifier; and

computer executable instructions for writing the subset of the data objects in the data container out to physical storage in response to the occurrence of an event, wherein the computer program product further comprises;

computer executable instructions for receiving a modification of a retention value of a data object of the subset of the data objects;

computer executable instructions for determining whether the data object of the subset of the data objects is to be moved to another data container; and

computer executable instructions in response to determining that the data object of the subset of the data objects is to be moved to another data container, for associating the data object of the subset of the data objects with another data container containing other data objects that have a retention value that is similar to the modified retention value of the data object of the subset of the data objects.

16. The computer program product of claim 15, wherein each data object of the plurality of data objects is annotated, by an application at data object creation time, with the associated expected lifetime identifier.

17. The computer program product of claim 15, wherein the criteria are one of periods of time or relative priorities.

18. The computer program product of claim 16, wherein the associated modification function defines how the retention value changes over time.

19. The computer program product of claim 18, wherein the modification function is a decay function that reduces the retention value over time;

and

wherein as the retention value increases, the amount of time to reduce the retention value increases by an order of magnitude.

20. The computer program product of claim 15, further comprising:

computer executable instructions for determining if an expected lifetime identifier of a data object of the subset of the data objects in the data container has a predetermined relationship with a deletion threshold; and

computer executable instructions for deleting all of the data objects of the subset of the data objects in the data container, in response to determining that the expected lifetime identifier of the data object of the subset of the data objects in the data container has the predetermined relationship with the deletion threshold.

21. The computer program product of claim 20, wherein the predetermined relationship is that the expected lifetime identifier has a value that is less than or equal to the value of the deletion threshold, and further comprising:

computer executable instructions for dynamically updating a value of the deletion threshold based on a current utilization of the storage system, wherein the sixth instructions for dynamically updating a value of the deletion threshold includes;

computer executable instructions for determining a current level of usage of the storage system;

computer executable instructions for increasing the value of the deletion threshold if the current level of usage of the storage system indicates an increase in usage of the storage system; and

computer executable instructions for decreasing the value of the deletion threshold if the current level of usage of the storage system indicates a decrease in usage of the storage system.

22. The computer program product of claim 18, wherein each of the expected lifetime identifiers and each of the associated modification functions are maintained in a metadata data structure in memory after writing of the subset of the data objects in the data container out to physical storage, and wherein the data container is deleted from memory after writing of the subset of the data objects in the data container out to physical storage.

23. The computer program product of claim 15, wherein the event is one of the data container becoming full, an elapse of a predetermined period of time, and a flush of the data container.

24. The computer program product of claim 15, wherein the event is detecting that an expected lifetime identifier of a data object of the subset of the data objects in the data container has been changed a predetermined number of times or more in a predetermined period of time, and wherein the computer executable instructions for writing the data objects of the subset of the data objects in the data container out to physical storage in response to the occurrence of an event includes computer executable instructions for writing the data object of the subset of the data objects in the data container out to a different physical storage than data objects whose expected lifetime identifiers have not changed a predetermined number of times or more in a predetermined period of time.

26. The computer program product of claim 15, wherein the computer executable instructions for determining whether the data object of the subset of the data objects is to be moved to another data container, comprises:

computer executable instructions for determining whether the data object of the subset of the data objects is to be moved based on applying data container policies for handling modifications to attributes of data objects.

27. The computer program product of claim 15, and further comprising:

computer executable instructions in response to determining that the data object of the subset of the data objects is to be moved to another data container, for also marking the data object of the subset of the data objects in the data container for deletion.

28. The computer program product of claim 15, and further comprising:

computer executable instructions in response to determining that the data object of the subset of the data objects is to be moved to another data container, for determining whether to modify the retention value associated with the data container; and

computer executable instructions in response to determining that the retention value associated with the data container is to be modified, for updating the retention value associated with the data container.

29. The computer program product of claim 28, and further comprising:

computer executable instructions in response to determining that the data object of the subset of the data objects is not to be moved to another data container, for updating metadata for the data container.

25. A system for storing data, comprising:

a memory storing computer executable instructions; and

a processor executing the computer executable instructions for;

receiving a plurality of data objects, wherein each data object of the plurality of data objects has an associated expected lifetime identifier that identifies criteria indicative of an amount of time the associated data object is expected to be retained in the data storage system, wherein each expected lifetime identifier comprises a retention value that indicates a relative importance of retaining the associated data object and an associated modification function for modifying the retention value and wherein the retention value is selected from a range of numbers between and including a first real number and a second real number; and

wherein the first real number is a lowest number of the range of numbers indicating the associated data object is not to be retained and the second real number is a highest number of the range of numbers indicating the associated data object is not to be deleted;

grouping a subset of the data objects into a data container in a storage device based on the data objects having a similar expected lifetime identifier; and

writing the subset of the data objects in the data container out to physical storage in response to an occurrence of an event, wherein the processor further executes the instructions for;

receiving a modification of a retention value of a data object of the subset of the data objects;

determining whether the data object of the subset of the data objects is to be moved to another data container; and

in response to determining that the data object of the subset of the data objects is to be moved to another data container, associating the data object of the subset of the data objects with another data container containing other data objects that have a retention value that is similar to the modified retention value of the data object of the subset of the data objects.