System and method for optimizing a storage system to support short data lifetimes

• US 20060072400A1
• Filed: 09/17/2004
• Published: 04/06/2006
• Est. Priority Date: 09/17/2004
• Status: Active Grant

• Current Assignee: National Security Agency
• Original Assignee: International Business Machines Corporation
• Inventors: Tracey, John Michael, Douglis, Frederick, Halim, Nagui, Tetzlaff, William Harold, Richards, Elizabeth Suzanne, Palmer, John Davis, Tao, David, Wolf, Joel Leonard, Anderson, Kay Schwendimann
• Granted Patent: US 7,958,093 B2
• Time in Patent Office: Days
• US Class Current: 369/47.100
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

A system and method for optimizing a storage system to support short data object lifetimes and highly utilized storage space are provided. With the system and method, data objects are clustered based on when they are anticipated to be deleted. When an application stores data, the application provides an indicator of the expected lifetime of the data, which may be a retention value, a relative priority of the data object, or the like. Data objects having similar expected lifetimes are clustered together in common data structures so that clusters of objects may be deleted efficiently in a single operation. Expected lifetimes may be changed by applications automatically. The system automatically determines how to handle these changes in expected lifetime using one or more of copying the data object, reclassifying the container in which the data object is held, and ignoring the change in expected lifetime for a time to investigate further changes in expected lifetime of other data objects.

Claims

1. A method of storing data in a data storage system, comprising:

receiving a plurality of data objects, wherein the data objects have associated expected lifetime identifiers that identify criteria indicative of an amount of time the associated data object is expected to be maintained in the data storage system;

grouping a subset of the data objects into a data container in a storage device based on the data objects having a similar expected lifetime identifier; and

writing the data objects in the data container out to physical storage in response to the occurrence of an event.

2. The method of claim 1, wherein the data objects are annotated, by an application at data object creation time, with the expected lifetime identifiers.

3. The method of claim 1, wherein the criteria are periods of time.

4. The method of claim 1, wherein the criteria are relative priorities.

5. The method of claim 2, wherein the data objects are further annotated, by an application at data object creation time, with an expected lifetime identifier modification function.

6. The method of claim 5, wherein the expected lifetime identifier modification function is a decay function that reduces a value of the expected lifetime identifier over time.

7. The method of claim 1, further comprising:

determining if an expected lifetime identifier of a data object in the data container has a predetermined relationship with a deletion threshold; and

deleting all of the data objects in the data container, if the expected lifetime identifier of the data object in the data container has the predetermined relationship with the deletion threshold.

8. The method of claim 7, wherein the predetermined relationship is that the expected lifetime identifier has a value that is less than or equal to a value of the deletion threshold.

9. The method of claim 7, further comprising:

dynamically updating a value of the deletion threshold based on a current utilization of the storage system.

10. The method of claim 9, wherein the predetermined relationship is that the expected lifetime identifier has a value that is less than or equal to the value of the deletion threshold, and wherein dynamically updating a value of the deletion threshold includes:

determining a current level of usage of the storage system;

increasing the value of the deletion threshold if the current level of usage of the storage system indicates an increase in usage of the storage system; and

decreasing the value of the deletion threshold if the current level of usage of the storage system indicates a decrease in usage of the storage system.

11. The method of claim 5, wherein the expected lifetime identifiers and expected lifetime identifier modification functions are maintained in a metadata data structure in memory after writing of the data container to physical storage, and wherein the data container is deleted from memory after writing of the data container to physical storage.

12. The method of claim 1, wherein the event is one of the data container becoming full, an elapse of a predetermined period of time, and a flush of the data container.

13. The method of claim 1, wherein the event is determining that the expected lifetime identifiers of the data objects indicate that the data objects are not to be deleted from the physical storage, and wherein writing the data objects in the data container out to physical storage in response to the occurrence of an event includes writing the data objects in the data container to a different physical storage than data objects whose expected lifetime identifiers indicate that the data objects are to be deleted from physical storage.

14. The method of claim 1, wherein the event is detecting that a expected lifetime identifier of a data object in the data container has been changed a predetermined number of times or more in a predetermined period of time, and wherein writing the data objects in the data container out to physical storage in response to the occurrence of an event includes writing the data objects in the data container to a different physical storage than data objects whose expected lifetime identifiers have not changed a predetermined number of times or more in a predetermined period of time.

16. The computer program product of claim 13, wherein the data objects are annotated, by an application at data object creation time, with the expected lifetime identifiers.

19. The computer program product of claim 16, wherein the data objects are further annotated, by an application at data object creation time, with an expected lifetime identifier modification function.

20. The computer program product of claim 19, wherein the expected lifetime identifier modification function is a decay function that reduces a value of the expected lifetime identifier over time.

25. The computer program product of claim 19, wherein the expected lifetime identifiers and expected lifetime identifier modification functions are maintained in a metadata data structure in memory after writing of the data container to physical storage, and wherein the data container is deleted from memory after writing of the data container to physical storage.

15. A computer program product in a computer readable medium for storing data in a data storage system, comprising:

first instructions for receiving a plurality of data objects, wherein the data objects have associated expected lifetime identifiers that identify criteria indicative of an amount of time that the associated data object is expected to be retained in the data storage system;

second instructions for grouping a subset of the data objects into a data container in a storage device based on the data objects having a similar expected lifetime identifier; and

third instructions for writing the data objects in the data container out to physical storage in response to the occurrence of an event.

17. The computer program product of claim 15, wherein the criteria are periods of time.

18. The computer program product of claim 15, wherein the criteria are relative priorities.

21. The computer program product of claim 15, further comprising:

fourth instructions for determining if an expected lifetime identifier of a data object in the data container has a predetermined relationship with a deletion threshold; and

fifth instructions for deleting all of the data objects in the data container, if the expected lifetime identifier of the data object in the data container has the predetermined relationship with the deletion threshold.

22. The computer program product of claim 21, wherein the predetermined relationship is that the expected lifetime identifier has a value that is less than or equal to a value of the deletion threshold.

23. The computer program product of claim 21, further comprising:

sixth instructions for dynamically updating a value of the deletion threshold based on a current utilization of the storage system.

24. The computer program product of claim 23, wherein the predetermined relationship is that the expected lifetime identifier has a value that is less than or equal to the value of the deletion threshold, and wherein the sixth instructions for dynamically updating a value of the deletion threshold include:

instructions for determining a current level of usage of the storage system;

instructions for increasing the value of the deletion threshold if the current level of usage of the storage system indicates an increase in usage of the storage system; and

instructions for decreasing the value of the deletion threshold if the current level of usage of the storage system indicates a decrease in usage of the storage system.

26. The computer program product of claim 15, wherein the event is one of the data container becoming full, an elapse of a predetermined period of time, and a flush of the data container.

27. The computer program product of claim 15, wherein the event is determining that the expected lifetime identifiers of the data objects indicate that the data objects are not to be deleted from the physical storage, and wherein the third instructions for writing the data objects in the data container out to physical storage in response to the occurrence of an event include instructions for writing the data objects in the data container to a different physical storage than data objects whose expected lifetime identifiers indicate that the data objects are to be deleted from physical storage.

28. The computer program product of claim 15, wherein the event is detecting that a expected lifetime identifier of a data object in the data container has been changed a predetermined number of times or more in a predetermined period of time, and wherein third instructions for writing the data objects in the data container out to physical storage in response to the occurrence of an event include instructions for writing the data objects in the data container to a different physical storage than data objects whose expected lifetime identifiers have not changed a predetermined number of times or more in a predetermined period of time.

29. A system for storing data, comprising:

means for receiving a plurality of data objects, wherein the data objects have associated expected lifetime identifiers that indicative of an amount of time that the associated data object is expected to be retained in a data storage system;

means for grouping a subset of the data objects into a data container in a storage device based on the data objects having a similar expected lifetime identifier; and

means for writing the data objects in the data container out to a contiguous portion of physical storage in response to the occurrence of an event.