Method and apparatus for optimizing data storage

• US 8,996,480 B2
• Filed: 05/04/2011
• Issued: 03/31/2015
• Est. Priority Date: 05/04/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Agarwala, Sandip, Bathen, Luis Angel D., Jadav, Divyesh
• Primary Examiner(s): Arjomandi, Noosha
• Application Number:
  US13/100,323
• Publication Number:
  US 20120284239A1
• Time in Patent Office: 1,427 Days
• Field of Search: 707/693, 707/812
• US Class Current: 707/693
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

Embodiments of the invention relate to evaluation and storage of data in a computer system configured with a shared pool of resources. A multi-level adaptive compression technique is employed to minimize the cost of data storage based upon the type of data being stored and their access pattern. The costs of data storage include capacity, bandwidth, and compute cycles. Data is transformed local to a client in communication with the shared pool, local to the shared pool, or as a combination with a partial transformation local to the client and a partial transformation local to the shared pool.

Claims

1. A method comprising:

processing application data for storage;

evaluating the processed data including using a real-time analytical model for recommending a compression technique together with a storage location, the evaluation including minimizing a cost for storage of the data the cost incorporating options of local compression, data transfer, and data storage;

the recommending includes determining whether to compress the data, and when compression is unwarranted sending the data to a first selected location, when compression is warranted determining an appropriate compression technique that satisfies the analytical model evaluation, compressing the data based upon the appropriate compression technique, and sending the compressed data to a second selected storage location, wherein the first and second selected storage locations are a part of a shared pool of resources and selected from the group consisting of;

different storage locations in the shared pool, and the same storage location in the shared pool.

2. The method of claim 1, wherein evaluation of the data includes an application selecting and applying a compression technique to the data, the application is in a location with respect to a shared pool of resources selected from the group consisting of:

local, remote, and a combination thereof.

3. The method of claim 2, further comprising the local application and the remote application cooperatively achieving compression of the evaluated data.

4. The method of claim 1, further comprising profiling each prior data format selection and data storage location, and quantifying a cost component to the profile data and storing the profile in a data structure.

5. The method of claim 4, wherein the quantified cost component includes a compression ratio of the data, storage location of data, and retrieval costs of the stored data.

6. The method of claim 1, the overall cost associated with storage of the data, includes a cost of transferring the data between a client machine and the shared pool of configurable computer resources and compressing and decompressing the data.

7. A system comprising:

a client machine with a processing unit and in communication with application data;

an evaluation manager in communication with the processing unit, the evaluation manager to process the application data, including use of a real-time analytical model for recommendation of a compression technique together with a storage location, the evaluation manager to minimize a cost for storage of the data, the cost incorporating options of local compression, data transfer, and data storage;

a compression manager in communication with the evaluation manager, the compression manager to determine whether to compress the data, and when compression is unwarranted to send the data to a first selected location, when compression is warranted to determine an appropriate compression technique that satisfies the analytical model evaluation, compress the data with the appropriate compression technique, and to send the compressed data to a second selected storage location, wherein the first and second selected storage locations are a part of a shared pool of resources and selected from the group consisting of;

different storage locations in the shared pool, and the same storage location in the shared pool.

8. The system of claim 7, further comprising the compression manager to select and apply a compression technique to the data, and wherein the compression manager is in a location with respect to a shared pool of resources selected from the group consisting of:

local, remote, and a combination thereof.

9. The system of claim 8, further comprising a first application local to the client machine and a second application remote from the client machine to cooperatively compress the evaluated data.

10. The system of claim 7, further comprising a profile manager in communication with the evaluation manager, the profile manager to profile each prior data format selection and data storage location, and to quantify a cost component to the profiled data and to store the profile in a data structure.

11. The system of claim 10, further comprising the evaluation manager to employ the quantified cost component of one or more stored profiles as a factor in evaluation of the processed data and selection of a storage format and location.

12. The system of claim 10, wherein the quantified cost component includes a compression ratio of the data, storage location of data, and retrieval costs of the stored data.

13. The system of claim 7, wherein the overall cost associated with storage of the data, includes a cost of transferring the data between the client machine and a shared pool of configurable computer resources and a cost of compressing and decompressing the data.

14. A computer program product, the computer program product comprising a non-transitory computer readable storage device having computer readable program code embodied therewith, the computer readable program code comprising:

computer readable program code configured to process data, including use of a real-time analytical model for recommending a compression technique together with a storage location, the recommendation based on minimization of cost for storage of the data, the cost incorporating options of local compression, data transfer, and data storage;

computer readable program code configured to determine whether to compress the data, and when compression is unwarranted to send the data to a first selected storage location, when compression is warranted to determine an appropriate compression technique that satisfies the analytical model and computer readable program code to send the compressed data to a second selected storage location, wherein the first and second selected storage locations are a part of a shared pool of resources and selected from the group consisting of;

different storage locations in the shared pool, and the same storage location in the shared pool.

15. The computer program product of claim 14, further comprising computer readable program code to select and apply a compression technique to the data, and wherein the program code is delivered as a service over a network and resides in a location with respect to a shared pool of resources selected from the group consisting of:

local, remote, and a combination thereof.

16. The computer program product of claim 15, further comprising a first application of program code local to a client machine in communication with the shared pool of resources over the network and a second application of program code remote from the client machine, the first and second applications of program code to cooperatively compress the evaluated data.

17. The computer program product of claim 14, further comprising computer readable program code to profile each prior data format selection and data storage location, and to quantify a cost component to the profiled data and to store the profile in a data structure.

18. The computer program product of claim 17, further comprising computer readable program code to employ the quantified cost component of one or more stored profiles as a factor in evaluation of the processed data and selection of a storage format and location, and wherein the quantified cost component includes a compression ratio of the data, storage location of data, and retrieval costs of the stored data.

19. The computer program product of claim 14, wherein the overall cost associated with storage of the data, includes a cost of transferring the data between the client machine and a shared pool of configurable computer resources and a cost of compressing and decompressing the data.