Systems and methods for transformation of logical data objects for storage

• US 9,176,976 B2
• Filed: 07/21/2011
• Issued: 11/03/2015
• Est. Priority Date: 05/31/2006
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Koifman, Chaim, Kedem, Nadav, Zohar, Avi, Amit, Jonathan
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Wong, Huen
• Application Number:
  US13/188,433
• Publication Number:
  US 20110276545A1
• Time in Patent Office: 1,566 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 12/023 Free address space management
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 2212/1044 Space efficiency improvement
• CPC Class Codes: G06F 2212/152 Virtualized environment, e.g. logically partitioned system
• CPC Class Codes: G06F 2212/401 Compressed data
• CPC Class Codes: G06F 2212/402 Encrypted data
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0623 in relation to content
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0661 : Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/0665 : at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0673 : Single storage device
• CPC Class Codes: H03M 7/30 : Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction

Abstract

Systems and methods for compressing a raw logical data object (201) for storage in a storage device operable with at least one storage protocol, creating, reading, writing, optimizatic in and restoring thereof. Compressing the raw logical data object (201) comprises creating in the storage device a compressed logical data object (203) comprising a header (204) and one or more allocated compressed sections with predefined size (205-1-205-2); compressing one or more sequentially obtained chunks of raw data (202-1-202-6) corresponding to the raw logical data object (201) thus giving rise to the compressed data chunks (207-1-207-6); and sequentially accommodating the processed data chunks into: said compressed sections (205-1-205-2) in accordance with an order said chunks received, wherein said compressed sections serve as atomic elements of compression/decompression operations during input/output transactions on the logical data object.

Claims

1. A method for writing data to a storage device configured to operate with at least one storage protocol, said method comprising:

generating, in the storage device, a compressed logical data object comprising at least one allocated section with a predefined size;

receiving variable size chunks of raw data;

comparing a characteristic of the raw data chunks to a predefined criterion related to an estimated or actual time of compression of the raw data chunks, the predefined criterion including an estimated time necessary to compress the raw data chunks which, being compressed, would substantially amount to a compressed section size;

if the characteristic matches the predefined criterion;

sequentially compressing the raw data chunks in accordance with an order said raw data chunks are received to generate compressed data chunks, wherein at least two of the compressed data chunks comprise different sizes,

accumulating the compressed data chunks in the at least one allocated section, the compressed data chunks accumulated until the at least one allocated section includes an amount of compressed data chunks matching the predefined size, and

writing said accumulated compressed data chunks into said at least one allocated section in accordance with an order said chunks of raw data are received; and

if the characteristic does not match the predefined criterion;

identifying the raw data chunks as being in a non-compressed form or a partly compressed form,

accumulating the non-compressed or partly compressed data chunks in the at least one allocated section, the non-compressed or partly compressed data chunks accumulated until the at least one allocated section includes an amount of data chunks matching the predefined size, and

writing said accumulated non-compressed or partly compressed data chunks into said at least one allocated section in accordance with an order said chunks of raw data are received.

2. The method of claim 1, further comprising mapping the data in the compressed logical data object to the data stored in each compressed section.

3. The method of claim 1, wherein the predefined size is equal for each compressed section.

4. The method of claim 1, wherein the size of the compressed section is configurable in accordance with a predetermined criterion.

5. The method of claim 1, wherein compressing the variable size chunks of raw data is performed via an adaptive dictionary, and the data chunks are stored in a same compressed section via a same dictionary.

6. The method of claim 1, wherein compressing the variable size chunks of raw data is performed via a plurality of adaptive dictionaries, and at least two data chunks are stored in different compressed sections via different dictionaries.

7. The method of claim 1, wherein each compressed section is associated with a flag indicating a use/re-use condition of a respective physical location of the compressed section, and each data chunk is associated with a same flag as the compressed section storing each data chunk.

8. A system for compressing a raw logical data object for storage in a storage device configured to operate with at least one storage protocol, said system comprising:

a processor; and

memory coupled to said processor, wherein the processor is configured to store computer code that, when executed by the processor, causes the processor to;

generate, in the storage device, a compressed logical data object comprising at least one allocated section with a predefined size,

receive variable size chunks of raw data,

compare a characteristic of the raw data chunks to a predefined criterion related to an estimated or actual time of compression of the raw data chunks, the predefined criterion including an estimated time necessary to compress the raw data chunks which, being compressed, would substantially amount to a compressed section size;

if the characteristic matches the predefined criterion;

sequentially compress the raw data chunks in accordance with an order said raw data chunks are received to generate compressed data chunks, wherein at least two of the compressed data chunks comprise different sizes,

accumulate the compressed data chunks in the at least one allocated section, the compressed data chunks accumulated until the at least one allocated section includes an amount of compressed data chunks matching the predefined size, and

write said accumulated compressed data chunks into said at least one allocated section in accordance with an order said chunks of raw data are received, and

if the characteristic does not match the predefined criterion;

identify the raw data chunks as being in a non-compressed form or a partly compressed form,

accumulate the non-compressed or partly compressed data chunks in the at least one allocated section, the non-compressed or partly compressed data chunks accumulated until the at least one allocated section includes an amount of data chunks matching the predefined size, and

write said accumulated non-compressed or partly compressed data chunks into said at least one allocated section in accordance with an order said chunks of raw data are received.

9. The system of claim 8, wherein the processor is further configured to map the data in the compressed logical data object to the data stored in each compressed section.

10. The system of claim 8, wherein the predefined size is equal for all compressed sections.

11. The system of claim 8, wherein the size of the compressed section is configurable in accordance with a predetermined criterion.

12. The system of claim 8, further comprising an adaptive dictionary, wherein compression is provided via an adaptive dictionary, and the data chunks are stored in a same compressed section via a same dictionary.

13. The system of claim 8, further comprising a plurality of adaptive dictionaries, wherein compression is provided via an adaptive dictionary, and at least two data chunks are stored in different compressed sections via different dictionaries.

14. The system of claim 8, wherein each compressed section is associated with a flag indicating a use/re-use condition of a respective physical location of the compressed section, and each data chunk is associated with a same flag as the compressed section storing each data chunk.

15. A computer program product comprising a computer useable medium including a computer readable program code embodied therein for compressing a raw logical data object for storage in a storage device configured to operate with at least one storage protocol, said computer program product comprising:

computer code for generating, in the storage device, a compressed logical data object comprising at least one allocated section with a predefined size;

computer code for receiving variable size chunks of raw data;

computer code for comparing a characteristic of the raw data chunks to a predefined criterion related to an estimated or actual time of compression of the raw data chunks, the predefined criterion including an estimated time necessary to compress the raw data chunks which, being compressed, would substantially amount to a compressed section size;

if the characteristic matches the predefined criterion;

computer code for sequentially compressing the raw data chunks in accordance with an order said raw data chunks are received to generate compressed data chunks, wherein at least two of the compressed data chunks comprise different sizes,

computer code for accumulating the compressed data chunks in the at least one allocated section, the compressed data chunks accumulated until the at least one allocated section includes an amount of compressed data chunks matching the predefined size, and

computer code for writing said accumulated compressed data chunks into said at least one allocated section in accordance with an order said chunks of raw data are received; and

if the characteristic does not match the predefined criterion;

computer code for identifying the raw data chunks as being in a non-compressed form or a partly compressed form,

computer code for accumulating the non-compressed or partly compressed data chunks in the at least one allocated section, the non-compressed or partly compressed data chunks accumulated until the at least one allocated section includes an amount of data chunks matching the predefined size, and

computer code for writing said accumulated non-compressed or partly compressed data chunks into said at least one allocated section in accordance with an order said chunks of raw data are received.

16. The computer program product of claim 15, further comprising computer code for mapping the data in the compressed logical data object to the data stored in each compressed section.

17. The computer program product of claim 15, wherein the predefined size is equal for all compressed sections.

18. The computer program product of claim 15, wherein the size of the compressed section is configurable in accordance with predetermined criterion.

19. The computer program product of claim 15, wherein the compression is provided via an adaptive dictionary, and the data chunks are stored in a same compressed section via a same dictionary.

20. The computer program product of claim 15, wherein the compression is provided via a plurality of adaptive dictionaries, and at least two data chunks are stored in different compressed sections via different dictionaries.

21. The computer program product of claim 15, wherein each compressed section is associated with a flag indicating a use/re-use condition of a respective physical location of the compressed section, and each data chunk is associated with a same flag as the compressed section storing each data chunk.