Systems and methods for transformation of logical data objects for storage

• US 9,479,616 B2
• Filed: 10/23/2015
• Issued: 10/25/2016
• Est. Priority Date: 05/31/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Koifman, Chaim, Kedem, Nadav, Zohar, Avi
• Primary Examiner(s): Truong, Thanhnga B
• Application Number:
  US14/921,319
• Publication Number:
  US 20160050297A1
• Time in Patent Office: 368 Days
• Field of Search: 713/181, 713/189, 713/193, 707/693, 707/999.001, 707/999.2, 709/213, 709/231, 380/37, 380/269
• US Class Current: 1/1
• CPC Class Codes: G06F 12/1408 by using cryptography for digital transmission H04L9/00
• CPC Class Codes: G06F 16/116 Details of conversion of file system types or formats
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: G06F 2212/1052 Security improvement
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC
• CPC Class Codes: H04L 9/0637 Modes of operation, e.g. cipher block chaining [CBC], electronic codebook [ECB] or Galois/counter mode [GCM]
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

Methods and systems for compressing a logical data object for storage in a storage device in a distributed network are provided. One method includes creating, in the storage device, a compressed logical data object including a header and one or more allocated compressed sections with a predefined size, compressing the one or more obtained chunks of data corresponding to the logical data object, thus giving rise to the compressed data chunks, and storing the compressed data chunks in the compressed sections, wherein the compressed sections serve as atomic elements of compression or decompression operations during input/output transactions on the logical data object.

Claims

1. A method of transforming a logical data object for storage in a hardware storage device in a distributed network of computing devices, comprising:

creating, in the hardware storage device within the distributed network of computing devices, a transformed logical data object comprising a header and a plurality of storage sections with a predefined size;

obtaining a plurality of data chunks corresponding to the transformed logical data object;

determining if the plurality of obtained data chunks exceed a predetermined threshold based on one of a time to compress each obtained data chunk and an amount of available storage space in a particular storage section that is dependent upon whether the particular storage section includes sufficient storage space to store a next uncompressed data chunk queued for storage;

compressing each obtained data chunk if the predetermined threshold based on time is not exceeded and not compressing each obtained data chunk if the predetermined threshold based on time is exceeded;

compressing each obtained data chunk if the predetermined threshold based on the amount of available storage space is exceeded and not compressing each obtained data chunk if the predetermined threshold based on the amount of available storage space is not exceeded, wherein compressing comprises;

compressing each obtained data chunk, and

accumulating the compressed data chunks to form compressed blocks, the compressed data chunks accumulated until a size of a compressed block includes the predefined size;

storing each compressed block into said storage sections; and

mapping between the data in the logical data object and the data stored in the storage sections.

2. The method of claim 1, wherein said compressing further comprises applying one of an encryption technique, an encoding technique, a conversion technique, and a combination thereof to each obtained data chunk when the predetermined threshold is not exceeded.

3. The method of claim 2, wherein:

the compressed obtained data chunk comprises a first portion of compressed data and a second portion of uncompressed data if the predetermined threshold is exceeded in relation to part of techniques of data compression; and

said first portion is provided by techniques related to the predetermined threshold.

4. The method of claim 1, wherein the predetermined threshold is related to at least one characteristic selected from a group comprising characteristics of the logical data object, characteristics of each obtained data chunk, characteristics of each storage section, characteristics of a transforming operation, and a combination thereof.

5. The method of claim 1, wherein the header of the compressed logical data object comprises a unique descriptor of the transformed logical data object, information related to a size of the compressed logical data object, and an indication if the compressed logical data object holds data in an uncompressed form.

6. The method of claim 1, wherein:

the compressed data chunks are stored in a log form; and

a log of a particular obtained data chunk comprises a log header containing information in respect of an offset of the particular obtained data chunk within the compressed logical data object, a size of said particular obtained data chunk, an identifier allowing associating the log with the storage section storing the log, and an indication of compressed/uncompressed form of data held in the log.

7. The method of claim 1, wherein the mapping is provided with a help of at least one index section constituting a part of the compressed logical data object, said index section comprising at least one entry holding at least information related to compressed obtained data chunks stored in at least one storage section, an indication of compressed/uncompressed form of data held in said obtained data chunks, and indication of physical storage location pertaining to said storage section.

8. A distributed network of computing devices for transforming a logical data object, comprising:

a hardware storage device within the distributed network of computing devices;

a processor coupled to the hardware storage device, the processor configured for;

creating in the hardware storage device a transformed logical data object comprising a header and a plurality of storage sections with a predefined size,

obtaining a plurality of data chunks corresponding to the transformed logical data object,

determining if the plurality of obtained data chunks exceed a predetermined threshold based on one of a time to compress each obtained data chunk and an amount of available storage space in a particular storage section that is dependent upon whether the particular storage section includes sufficient storage space to store a next uncompressed data chunk queued for storage,

compressing each obtained data chunk if the predetermined threshold based on time is not exceeded and not compressing each obtained data chunk if the predetermined threshold based on time is exceeded,

compressing each obtained data chunk if the predetermined threshold based on the amount of available storage space is exceeded and not compressing each obtained data chunk if the predetermined threshold based on the amount of available storage space is not exceeded, wherein compressing comprises;

compressing each obtained data chunk, and

accumulating the compressed data chunks to form compressed blocks, the compressed data chunks accumulated until a size of a compressed block includes the predefined size;

storing each compressed block into said storage sections; and

mapping between the data in the logical data object and the data stored in the storage sections.

9. The distributed network of claim 8, wherein, when the predetermined threshold is not exceeded, the processor is configured for applying one of an encryption technique, an encoding technique, a conversion technique, and a combination thereof to each obtained data chunk.

10. The distributed network of claim 9, wherein:

the compressed obtained data chunk comprises a first portion of compressed data and a second portion of uncompressed data when the predetermined threshold is exceeded in relation to part of techniques of data compression, and

said first portion is provided by techniques related to the predetermined threshold.

11. The distributed network of claim 8, wherein the predetermined threshold is related to at least one characteristic selected from a group comprising characteristics of the logical data object, characteristics of each obtained data chunk, characteristics of each storage section, characteristics of a transforming operation, and a combination thereof.

12. The distributed network of claim 8, wherein the header of the compressed logical data object comprises a unique descriptor of the transformed logical data object, information related to a size of the compressed logical data object, and an indication if the compressed logical data object holds data in an uncompressed form.

13. The distributed network of claim 8, wherein:

the compressed data chunks are stored in a log form; and

a log of a particular obtained data chunk comprises a log header containing information in respect of an offset of the particular obtained data chunk within the compressed logical data object, a size of said particular obtained data chunk, an identifier allowing associating the log with the storage section storing the log, and an indication of compressed/uncompressed form of data held in the log.

14. The distributed network of claim 8, wherein, when mapping, the processor is configured for utilizing at least one index section constituting a part of the compressed logical data object, said index section comprising at least one entry holding at least information related to compressed obtained data chunks stored in at least one storage section, an indication of compressed form or uncompressed form of data held in said obtained data chunks, and indication of physical storage location pertaining to said storage section.

15. A computer program product comprising a computer-readable memory having computer readable program code embodied therein for transforming a logical data object for storage in a hardware storage device in a distributed network of computing devices, said computer program product comprising:

computer code for creating, in the hardware storage device within the distributed network of computing devices, a transformed logical data object comprising a header and a plurality of storage sections with a predefined size;

computer code for obtaining a plurality of data chunks corresponding to the transformed logical data object;

computer code for determining if the plurality of obtained data chunks exceed a predetermined threshold based on one of a time to compress each obtained data chunk and an amount of available storage space in a particular storage section that is dependent upon whether the particular storage section includes sufficient storage space to store a next uncompressed data chunk queued for storage;

computer code for compressing each obtained data chunk if the predetermined threshold based on time is not exceeded and not compressing each obtained data chunk if the predetermined threshold based on time is exceeded;

computer code for compressing each obtained data chunk if the predetermined threshold based on the amount of available storage space is exceeded and not compressing each obtained data chunk if the predetermined threshold based on the amount of available storage space is not exceeded, wherein compressing comprises;

computer code for compressing each obtained data chunk, and

computer code for accumulating the compressed data chunks to form compressed blocks, the compressed data chunks accumulated until a size of a compressed block includes the predefined size;

computer code for storing each compressed block into said storage sections; and

computer code for mapping between the data in the logical data object and the data stored in the storage sections.

16. The computer program product of claim 15, further comprising computer code for applying one of an encryption technique, an encoding technique, a conversion technique, and a combination thereof to each obtained data chunk when the predetermined threshold is not exceeded, wherein:

the compressed obtained data chunk comprises a first portion of compressed data and a second portion of uncompressed data when the predetermined threshold is exceeded in relation to part of techniques of data compression, and

said first portion is provided by techniques related to the predetermined threshold.

17. The computer program product of claim 15, wherein the predetermined threshold is related to at least one characteristic selected from a group comprising characteristics of the logical data object, characteristics of each obtained data chunk, characteristics of each storage section, characteristics of a transforming operation, and a combination thereof.

18. The computer program product of claim 15, wherein the header of the compressed logical data object comprises a unique descriptor of the transformed logical data object, information related to a size of the compressed logical data object, and an indication if the compressed logical data object holds data in an uncompressed form.

19. The computer program product of claim 15, wherein:

the compressed data chunks are stored in a log form; and

a log of a particular obtained data chunk comprises a log header containing information in respect of an offset of the particular obtained data chunk within the compressed logical data object, a size of said particular obtained data chunk, an identifier allowing associating the log with the storage section storing the log, and an indication of compressed/uncompressed form of data held in the log.

20. The computer program product of claim 15, further comprising:

computer code for utilizing at least one index section constituting a part of the compressed logical data object, said index section comprising at least one entry holding at least information related to compressed obtained data chunks stored in at least one storage section, an indication of compressed form or uncompressed form of data held in said obtained data chunks, and indication of physical storage location pertaining to said storage section.