Systems and methods for transformation of logical data objects for storage

• US 9,110,913 B2
• Filed: 07/07/2011
• Issued: 08/18/2015
• Est. Priority Date: 05/31/2006
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Koifman, Chaim, Kedem, Nadav, Zohar, Avi
• Primary Examiner(s): Lee, Wilson
• Assistant Examiner(s): Wong, Huen
• Application Number:
  US13/178,418
• Publication Number:
  US 20110264871A1
• Time in Patent Office: 1,503 Days
• Field of Search: 707/687, 707/693
• US Class Current: 1/1
• CPC Class Codes: G06F 12/023 Free address space management
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 21/78 to assure secure storage of data address-based protection against unauthorised use of memory G06F12/14; record carriers for use with machines and with at least a part designed to carry digital markings G06K19/00
• CPC Class Codes: G06F 2212/1044 Space efficiency improvement
• CPC Class Codes: G06F 2212/152 Virtualized environment, e.g. logically partitioned system
• CPC Class Codes: G06F 2212/401 Compressed data
• CPC Class Codes: G06F 2212/402 Encrypted data
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0623 in relation to content
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0661 : Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/0665 : at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0673 : Single storage device
• CPC Class Codes: H03M 7/30 : Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction

Abstract

Systems and methods for transforming a logical data object for storage in a storage device operable with at least one storage protocol, creating, reading, writing, optimization and restoring thereof. Transforming the logical data object comprises creating in the storage device a transformed logical data object comprising one or more allocated storage sections with a predefined size; transforming one or more sequentially obtained chunks of obtained data corresponding to the transforming logical data object; and sequentially storing the processed data chunks into said storage sections in accordance with a receive order of said chunks, wherein said storage sections serve as atomic elements of transformation/de-transformation operations during input/output transactions on the logical data object. The processing may comprise two or more data transformation techniques coordinated in time, concurrently executing autonomous sets of instructions, and provided in a manner preserving the sequence of processing and storing the processed data chunks.

Claims

1. A method of transforming a logical data object for storage in a storage device coupled to a network utilizing a network protocol, said method comprising:

creating in the storage device a transformed logical data object comprising a plurality of allocated storage sections with a plurality of different predefined sizes, wherein each different predefined size corresponds to a different type of data stored in each storage section to create different sized storage sections;

receiving, via the network, a plurality of data chunks comprising a variable size and that correspond to the transformed logical data object at a transformation system separated from the storage device via the network;

comparing a characteristic of each received data chunk to a predefined criterion related to an estimated or actual time of transformation of each data chunk, the predefined criterion including an estimated time necessary to compress data which, being compressed, would substantially amount to a compressed section size;

transforming, by the transformation system, each of the plurality of data chunks, wherein transforming comprises compressing data chunks including text data, voice data, and image data at different ratios to fit in the different sized storage sections;

identifying a received data chunk as being in a non-transformed or partly transformed form and accumulating each non-transformed or partly transformed data chunk in one of the plurality of allocated storage sections, if the predefined criterion does not match the characteristic of the received data chunk, wherein the identification of the received data chunk as being in the non-transformed or partly transformed form is included as an entry in an index;

repeating the comparing and at least one of the transforming step and the identifying step for each subsequently received data chunk, and accumulating subsequent compressed data chunks and non-transformed data chunks or compressed data chunks and partly transformed data chunks to form an accommodation section having a size matching the different sized storage sections;

transmitting, via the network, the transformed plurality of data chunks to the storage device; and

sequentially storing the transformed plurality of data chunks into said plurality allocated storage sections in accordance with an order said plurality of data chunks are transformed and in accordance with the different type of data in each storage section,

wherein non-transformed and partly transformed data chunks are data chunks in which transformation was interrupted.

2. The method of claim 1, further comprising mapping data in the logical data object to the transformed data chunks stored in the one or more allocated storage sections.

3. The method of claim 1, wherein transforming comprises utilizing a plurality of data transformation techniques such that each of the data chunks comprises the transformed data.

4. The method of claim 3, wherein the plurality of data transformation techniques are coordinated in time, comprise concurrently executed autonomous sets of instructions, and are provided in a manner preserving a sequence of transforming and of storing the transformed data chunks.

5. The method of claim 1, wherein transforming comprises compressing the plurality of data chunks corresponding to the transformed logical data object resulting in a plaintext compressed data chunk, and further encrypting said plaintext compressed data chunks.

6. The method of claim 1, further comprising distinguishing between data chunks that are capable of being retrieved from each allocated storage section if being de-transformed and the data chunks stored in said allocated storage section as a result of said transforming.

7. A system for transforming a logical data object for storage in a storage device coupled to a network utilizing a network protocol, said system comprising:

a processor; and

memory coupled to the processor, wherein the memory is configured to store computer code that, when executed by the processor, causes the processor to;

create, in the storage device, a transformed logical data object comprising a plurality of allocated storage sections with a plurality of different predefined sizes, wherein each different predefined size corresponds to a different type of data stored in each storage section to create different sized storage sections,

receive, via the network, a plurality of data chunks comprising a variable size and that correspond to the transformed logical data object at a transformation system separated from the storage device via the network,

compare a characteristic of each received data chunk to a predefined criterion related to an estimated or actual time of transformation of each data chunk, the predefined criterion including an estimated time necessary to compress data which, being compressed, would substantially amount to a compressed section size;

transform, via the transformation system, each the plurality of data chunks, wherein transforming comprises compressing data chunks including text data, voice data, and image data at different ratios to fit in the different sized storage sections,

identify a received data chunk as being in a non-transformed or partly transformed form and accumulating each non-transformed or partly transformed data chunk in one of the plurality of allocated storage sections, if the predefined criterion does not match the characteristic of the received data chunk, wherein the identification of the received data chunk as being in the non-transformed or partly transformed form is included as an entry in an index,

repeat the comparing and at least one of the transforming step and the identifying step for each subsequently received data chunk, and accumulating subsequent compressed data chunks and non-transformed data chunks or compressed data chunks and partly transformed data chunks to form an accommodation section having a size matching the different sized storage sections,

transmit, via the network, the transformed plurality of data chunks to the storage device; and

sequentially store the transformed plurality of data chunks into said plurality of allocated storage sections in accordance with an order said transformed plurality of data chunks are transformed and in accordance with the different type of data in each storage section,

wherein non-transformed and partly transformed data chunks are data chunks in which transformation was interrupted.

8. The system of claim 7, wherein the memory is further configured to store computer code that, when executed by the processor, further causes the processor to map data in the logical data object to the transformed data chunks stored in the one or more allocated storage sections.

9. The system of claim 7, wherein the computer code that causes the processor to transform the data chunks comprises computer code that causes the processor to utilize a plurality of data transformation techniques such that each of the data chunks comprises the transformed data.

10. The system of claim 9, wherein the plurality of data transformation techniques are coordinated in time, comprise concurrently executed autonomous sets of instructions, and are provided in a manner preserving a sequence of transforming and of storing the transformed data chunks.

11. The system of claim 7, wherein the computer code that causes the processor to transform the data chunks comprises computer code that causes the processor to compress the plurality of data chunks corresponding to the transformed logical data object resulting in a plaintext compressed data chunk, and further computer code that causes the processor to encrypt said plaintext compressed data chunks.

12. The system of claim 7, wherein the memory is further configured to store computer code that, when executed by the processor, further causes the processor to distinguish between data chunks that are capable of being retrieved from each allocated storage section if being de-transformed and the data chunks stored in said allocated storage section as a result of being transformed.

13. A computer readable storage medium including a computer readable program code embodied therein for transforming a logical data object for storage in a storage device coupled to a network utilizing a network protocol, said computer program product comprising:

computer code for creating in the storage device a transformed logical data object comprising a plurality of allocated storage sections with a plurality of different predefined sizes, wherein each different predefined size corresponds to a different type of data stored in each storage section to create different sized storage sections;

computer code for receiving, via the network, a plurality of data chunks comprising a variable size and that correspond to the transformed logical data object at a transformation system separated from the storage device via the network;

computer code for comparing a characteristic of each received data chunk to a predefined criterion related to an estimated or actual time of transformation of each data chunk, the predefined criterion including an estimated time necessary to compress data which, being compressed, would substantially amount to a compressed section size;

computer code for transforming, via the transformation system, each of the plurality of data chunks, wherein transforming comprises compressing data chunks including text data, voice data, and image data at different ratios to fit in the different sized storage sections;

computer code for identifying a received data chunk as being in a non-transformed or partly transformed form and accumulating each non-transformed or partly transformed data chunk in one of the plurality of allocated storage sections, if the predefined criterion does not match the characteristic of the received data chunk, wherein the identification of the received data chunk as being in the non-transformed or partly transformed form is included as an entry in an index;

computer code for repeating the comparing and at least one of the transforming step and the identifying step for each subsequently received data chunk, and accumulating subsequent compressed data chunks and non-transformed data chunks or compressed data chunks and partly transformed data chunks to form an accommodation section having a size matching the different sized storage sections;

computer code for transmitting, via the network, the transformed plurality of data chunks to the storage device; and

computer code for sequentially storing the transformed plurality of data chunks into said plurality of allocated storage sections in accordance with an order said transformed plurality of data chunks are transformed and in accordance with the different type of data in each storage section,

wherein non-transformed and partly transformed data chunks are data chunks in which transformation was interrupted.

14. The computer readable storage medium of claim 13, further comprising computer code for mapping data in the logical data object to the transformed data chunks stored in the one or more allocated storage sections.

15. The computer readable storage medium of claim 13, wherein the computer code for transforming comprises computer code for utilizing a plurality of data transformation techniques such that each of the plurality of data chunks comprises the transformed data.

16. The computer readable storage medium of claim 15, wherein the plurality of data transformation techniques are coordinated in time, comprise concurrently executed autonomous sets of instructions, and are provided in a manner preserving a sequence of transforming and of storing the transformed plurality of data chunks.

17. The computer readable storage medium of claim 13, wherein the computer code for transforming comprises computer code for compressing each of the plurality of data chunks corresponding to the transformed logical data object resulting in a plaintext compressed data chunk, and further encrypting said plaintext compressed data chunks.

18. The computer readable storage medium of claim 13, further comprising computer code for distinguishing between data chunks that are capable of being retrieved from each allocated storage section if being de-transformed and the data chunks stored in said allocated storage section as a result of said transforming.