Systems and methods for transformation of logical data objects for storage

• US 8,954,403 B2
• Filed: 03/28/2012
• Issued: 02/10/2015
• Est. Priority Date: 05/31/2006
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Koifman, Chaim, Kedem, Nadav, Zohar, Avi
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Wong, Huen
• Application Number:
  US13/432,145
• Publication Number:
  US 20120191739A1
• Time in Patent Office: 1,049 Days
• Field of Search: 707/693, 707/687
• US Class Current: 707/693
• CPC Class Codes: G06F 12/023 Free address space management
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 2212/1044 Space efficiency improvement
• CPC Class Codes: G06F 2212/152 Virtualized environment, e.g. logically partitioned system
• CPC Class Codes: G06F 2212/401 Compressed data
• CPC Class Codes: G06F 2212/402 Encrypted data
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0623 in relation to content
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0661 : Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/0665 : at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0673 : Single storage device
• CPC Class Codes: H03M 7/30 : Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction

Abstract

Systems and methods for transforming a logical data object for storage in a storage device operable with at least one storage protocol, creating, reading, writing, optimization and restoring thereof. Transforming the logical data object comprises creating in the storage device a transformed logical data object comprising one or more allocated storage sections with a predefined size; transforming one or more sequentially obtained chunks of obtained data corresponding to the transforming logical data object; and sequentially storing the processed data chunks into said storage sections in accordance with a receive order of said chunks, wherein said storage sections serve as atomic elements of transformation/de-transformation operations during input/output transactions on the logical data object. The processing may comprise two or more data transformation techniques coordinated in time, concurrently executing autonomous sets of instructions, and provided in a manner preserving the sequence of processing and storing the processed data chunks.

Claims

1. A method for writing data to a storage device, said method comprising:

generating, in the storage device, a logical data object comprising at least one transformed section including a predefined size;

receiving variable size chunks of raw data;

comparing a characteristic of the variable size chunks of raw data to a predefined criterion related to an actual time of transformation or an estimated time of transformation, the estimated time of transformation including an estimated time necessary to transform data that, when transformed, would substantially amount to a transformed section size;

transforming the variable size chunks of raw data in accordance with an order said chunks are received to generate transformed data chunks, wherein at least two of the transformed data chunks comprise different sizes;

accumulating the transformed data chunks to form said at least one transformed section if the predefined criterion matches the characteristic of the variable size chunks of raw data;

identifying each of the variable size chunks of raw data as being non-transformed data or partially transformed data and accumulating the non-transformed data or the partially transformed data in the at least one transformed section if the predefined criterion does not match the characteristic, wherein the identification of the variable size chunks of raw data as being non-transformed data or partially transformed data is included as entries in an index; and

repeating the comparing step, repeating at least one of the transforming step and the identifying step for subsequently received variable size chunks of raw data, and accumulating subsequently transformed data chunks and non-transformed data chunks or subsequently transformed data chunks and partially transformed data chunks,

wherein;

the transformed data chunks are accumulated until the at least one transformed section includes a size matching the predefined size, and

accumulating comprises sequentially writing said transformed data chunks into said at least one transformed section in accordance with the order said chunks are received.

2. The method of claim 1, wherein the predefined size is equal for each transformed section.

3. The method of claim 1, wherein the size of the transformed section is configurable in accordance with a predetermined criterion.

4. The method of claim 1, wherein transforming comprises at least two of compressing, encrypting, encoding, and performing a conversion on the variable size chunks of raw data.

5. The method of claim 1, wherein transforming comprises at least three of compressing, encrypting, encoding, and performing a conversion on the variable size chunks of raw data.

6. The method of claim 1, wherein transforming comprises each of compressing, encrypting, encoding, and performing a conversion on the variable size chunks of raw data.

7. The method of claim 1, wherein each transformed section is associated with a flag indicating a use/re-use condition of a respective physical location of the transformed section, and each data chunk is associated with a same flag as the transformed section storing each data chunk.

8. A system for writing data, said system comprising:

a processor; and

a storage device coupled to said processor, wherein the storage device is configured to store computer code that, when executed by the processor, causes the processor to;

generate, in the storage device, a logical data object, the logical data object comprising at least one transformed section with a predefined size,

receive variable size chunks of raw data,

compare a characteristic of the variable size chunks of raw data to a predefined criterion related to an actual time of transformation or an estimated time of transformation, the estimated time of transformation including an estimated time necessary to transform data that, when transformed, would substantially amount to a transformed section size;

transform the variable size chunks of raw data in accordance with an order said chunks are received to generate transformed data chunks, wherein at least two of the transformed data chunks comprise different sizes,

accumulate the transformed data chunks to form said at least one transformed section if the predefined criterion matches the characteristic of the variable size chunks of raw data,

identify each of the variable size chunks of raw data as being non-transformed data or partially transformed data and accumulating the non-transformed data or the partially transformed data in the at least one transformed section if the predefined criterion does not match the characteristic, wherein the identification of the variable size chunks of raw data as being non-transformed data or partially transformed data is included as entries in an index, and

repeat the comparing step, repeat at least one of the transforming step and the identifying step for subsequently received variable size chunks of raw data, and accumulate subsequently transformed data chunks and non-transformed data chunks or subsequently transformed data chunks and partially transformed data chunks,

wherein;

the transformed data chunks are accumulated until the at least one transformed section includes a size matching the predefined size, and

accumulating comprises sequentially writing said transformed data chunks into said at least one transformed section in accordance with the order said chunks are received.

9. The system of claim 8, wherein the predefined size is equal for all transformed sections.

10. The system of claim 8, wherein the size of the transformed section is configurable in accordance with a predetermined criterion.

11. The system of claim 8, wherein, when transforming, the processor is configured to perform at least two of compressing, encrypting, encoding, and performing a conversion on the variable size chunks of raw data.

12. The system of claim 8, wherein, when transforming, the processor is configured to perform at least three of compressing, encrypting, encoding, and performing a conversion on the variable size chunks of raw data.

13. The system of claim 8, wherein, when transforming, the processor is configured to perform each of compressing, encrypting, encoding, and performing a conversion on the variable size chunks of raw data.

14. The system of claim 8, wherein each transformed section is associated with a flag indicating a use/re-use condition of a respective physical location of the transformed section, and each data chunk is associated with a same flag as the transformed section storing each data chunk.

15. A computer program product comprising a computer readable storage medium including a computer readable program code embodied therein for writing data to a storage device, said computer program product comprising:

computer code for generating, in the storage device, a logical data object comprising at least one transformed section with a predefined size;

computer code for receiving variable size chunks of raw data;

computer code for comparing a characteristic of the variable size chunks of raw data to a predefined criterion related to an actual time of transformation or an estimated time of transformation, the estimated time of transformation including an estimated time necessary to transform data that, when transformed, would substantially amount to a transformed section size;

computer code for transforming the variable size chunks of raw data in accordance with an order said chunks are received to generate transformed data chunks, wherein at least two of the transformed data chunks comprise different sizes;

computer code for accumulating the transformed data chunks to form said at least one transformed section if the predefined criterion matches the characteristic of the variable size chunks of raw data;

computer code for identifying each of the variable size chunks of raw data as being non-transformed data or partially transformed data and accumulating the non-transformed data or the partially transformed data in the at least one transformed section if the predefined criterion does not match the characteristic, wherein the identification of the variable size chunks of raw data as being non-transformed data or partially transformed data is included as entries in an index;

computer code for repeating the comparing step, computer code for repeating at least one of the transforming step and the identifying step for subsequently received variable size chunks of raw data, and computer code for accumulating subsequently transformed data chunks and non-transformed data chunks or subsequently transformed data chunks and partially transformed data chunks,

wherein;

the transformed data chunks are accumulated until the at least one transformed section includes a size matching the predefined size, and

accumulating comprises sequentially writing said transformed data chunks into said at least one transformed section in accordance with the order said chunks are received.

16. The computer program product of claim 15, wherein the predefined size is equal for all transformed sections.

17. The computer program product of claim 15, wherein the size of the transformed section is configurable in accordance with predetermined criterion.

18. The computer program product of claim 15, wherein the computer code for transforming the variable size chunks of raw data comprises computer code for performing at least two of compressing, encrypting, encoding, and performing a conversion on the variable size chunks of raw data.

19. The computer program product of claim 15, wherein the computer code for transforming the variable size chunks of raw data comprises computer code for performing at least three of compressing, encrypting, encoding, and performing a conversion on the variable size chunks of raw data.

20. The computer program product of claim 15, wherein the computer code for transforming the variable size chunks of raw data comprises computer code for performing each of compressing, encrypting, encoding, and performing a conversion on the variable size chunks of raw data.

21. The computer program product of claim 15, wherein each transformed section is associated with a flag indicating a use/re-use condition of a respective physical location of the transformed section, and each data chunk is associated with a same flag as the transformed section storing each data chunk.