EARLY COMPRESSION RELATED PROCESSING WITH OFFLINE COMPRESSION

• US 20170109367A1
• Filed: 10/16/2015
• Published: 04/20/2017
• Est. Priority Date: 10/16/2015
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Constantinescu, Mihail C., Glider, Joseph S., Harnik, Danny, Luan, Leo, Sawdon, Wayne A., Schmuck, Frank B.
• Application Number:
  US14/885,889
• Publication Number:
  US 20170109367A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files

Abstract

A method for early compression related processing in a file system with offline compression. The method includes receiving a data file in a buffer. A processor detects that at least a portion of a data block of the data file resides in the buffer. A compressibility indication of the data block is determined based on performing at least one compressibility analysis operation on the data block. The compressibility indication of the data block is stored. A background compression task is performed on the data block based on: determining a compression decision for the data block based on the compressibility indication, and compressing the data block based on the compression decision.

Claims

1. A method comprising:

receiving a data file in a buffer;

detecting, by a processor, that at least a portion of a data block of the data file resides in the buffer;

determining a compressibility indication of the data block based on performing at least one compressibility analysis operation on the data block;

storing the compressibility indication of the data block; and

performing a background compression task on the data block based on;

determine a compression decision for the data block based on the compressibility indication; and

compressing the data block based on the compression decision.

2. The method of claim 1, wherein receiving the data file comprises writing the data file and dirtying the data block as in-memory in the buffer, or reading the data file and filling the data block as in-memory in the buffer.

3. The method of claim 1, wherein the at least one compressibility analysis operation is configured to select a compression technique from a plurality of compression techniques or to skip compression of the data block.

4. The method of claim 1, wherein a compressibility analysis operation is performed on a data block only if such operation has not been performed on this block since its last update.

5. The method of claim 1, wherein the at least one operation comprises a sampling operation or an entropy estimation operation, and the compressibility indication is determined by comparing a result of the at least one operation to a system-defined threshold or a predetermined threshold.

6. The method of claim 1, wherein the compressibility indication is determined by categorizing the compressibility indication into a particular level of compressibility, the background compression task performs compression in an order based on level of compressibility, and the compressibility indication is stored in metadata associated with the data block.

7. The method of claim 1, wherein the compressibility indication is stored in the metadata based on one or more bits in a memory address field of the data block, or within a compressibility-bitmap attribute of the data file, and after the compressibility indication is determined, the data file and the metadata is flushed to a memory device.

8. The method of claim 7, wherein performing the background compression task further comprises:

opening a data file to be compressed;

obtaining per-block compressibility information from metadata associated with the data file to be compressed;

determining a compression decision for one or more file blocks of the data file to be compressed or for an aggregated group of file blocks of the data file to be compressed;

reading the one or more file blocks or the aggregated group of file blocks into one or more file system buffers;

compressing data of the one or more file blocks or the aggregated group of file blocks; and

writing the compressed data back to the memory device.

9. A computer program product for early compression related processing in a file system with offline compression, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:

obtain, by the processor, a data file in a buffer;

detect, by the processor, that at least a portion of a data block of the data file resides in the buffer;

determine, by the processor, a compressibility indication of the data block based on performing at least one compressibility analysis operation on the data block;

store, by the processor, the compressibility indication of the data block; and

perform, by the processor, a background compression task on the data block based on;

determining, by the processor, a compression decision for the data block based on the compressibility indication; and

compressing, by the processor, the data block based on the compression decision.

10. The computer program product of claim 9, wherein the processor obtains the data file by reading the data file and filling the data block as in-memory in the buffer, or writing the data file and dirtying the data block as in-memory in the buffer.

11. The computer program product of claim 9, wherein the at least one compressibility analysis operation is configured to select a compression technique from a plurality of compression techniques or to skip compression of the data block.

12. The computer program product of claim 11, wherein the compressibility indication is determined by the processor comparing a result of the at least one operation to a system-defined threshold or a predetermined threshold.

13. The computer program product of claim 9, wherein the compressibility indication is determined by the processor categorizing the compressibility indication into a particular level of compressibility, and the background compression task performs compression in an order based on level of compressibility.

14. The computer program product of claim 9, wherein the compressibility indication is stored in metadata of the data block, one or more bits in a memory address field of the data block, or within a compressibility-bitmap attribute of the data file, and after the compressibility indication is determined, the data file and the metadata is flushed to a memory device.

15. The computer program product of claim 14, wherein the background compression task further comprises program instructions executable by the processor to cause the processor to:

open, by the processor, a data file to be compressed;

obtain, by the processor, per-block compressibility information from metadata associated with the data file to be compressed;

determine, by the processor, a compression decision for one or more file blocks of the data file to be compressed or for an aggregated group of file blocks of the data file to be compressed;

read, by the processor, the one or more file blocks or the aggregated group of file blocks into one or more file system buffers;

compress, by the processor, data of the one or more file blocks or the aggregated group of file blocks; and

write, by the processor, the compressed data back to the memory device.

16. An apparatus comprising:

a buffer configured to store a data file;

a data analyzer processor configured to detect that at least a portion of a data block of the data file resides in the buffer, and to determine a compressibility indication of the data block based on performing at least one compressibility analysis operation on the data block;

a metadata processor configured to store the compressibility indication of the data block; and

a compression processor configured to perform a background compression task on the data block based on being configured to;

determine a compression decision for the data block based on the compressibility indication; and

compress the data block based on the compression decision.

17. The apparatus of claim 16, wherein the at least one operation comprises a sampling operation or an entropy estimation operation, and the data analyzer processor is configured to determine the compressibility indication by being configured to compare a result of the at least one operation to a system-defined threshold or a predetermined threshold.

18. The apparatus of claim 16, wherein the data analyzer processor is configured to determine the compressibility indication by being configured to categorize the compressibility indication into a particular level of compressibility, the compression processor is configured to perform compression in an order based on level of compressibility, the compressibility indication is stored in the metadata of the data block, one or more bits in a memory address field of the data block, or within a compressibility-bitmap attribute of the data file, and a storage processor is configured to flush the data file and the metadata to a memory device.

19. The apparatus of claim 18, wherein the compression processor is configured to:

open a data file to be compressed;

obtain per-block compressibility information from metadata associated with the data file to be compressed;

determine a compression decision for one or more file blocks of the data file to be compressed or for an aggregated group of file blocks of the data file to be compressed;

cause the storage processor to read the one or more file blocks or the aggregated group of file blocks into one or more file system buffers;

compress data of the one or more file blocks or the aggregated group of file blocks; and

cause the storage processor to write the compressed data back to the memory device.

20. The apparatus of claim 19, wherein the compression processor is configured to open the data file by one of:

writing the data file and dirtying the data block as in-memory in the buffer, or reading the data file and filling the data block as in-memory in the buffer, and to select a compression technique from a plurality of compression techniques or to skip compression of the data block.