Dual layer deduplication for application specific file types in an information processing system

• US 11,003,629 B2
• Filed: 10/31/2018
• Issued: 05/11/2021
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Alkalay, Amitai, Schneider, Zvi, Natanzon, Assaf
• Primary Examiner(s): Le, Hung D
• Application Number:
  US16/176,820
• Publication Number:
  US 20200134051A1
• Time in Patent Office: 923 Days
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0679 Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory. The at least one processing device is configured to obtain a file and to determine a type of the file. A structure of the file may be determined based at least in part on the determined type of the file and at least one location in the file may be identified based at least in part on the determined structure. The file may be divided at the identified at least one location into a plurality of chunks and the plurality of chunks may be provided to a block deduplication module of a storage system where the block deduplication module is configured to perform a deduplication process based at least in part on the plurality of chunks.

Claims

1. An apparatus comprising:

at least one processing device comprising a processor coupled to a memory, the at least one processing device being configured;

to obtain a file;

to determine a type of the file;

to determine a structure of the file based at least in part on the determined type of the file;

to identify at least one location in the file based at least in part on the determined structure;

to divide the file at the identified at least one location into a plurality of chunks; and

to provide the plurality of chunks to a block deduplication module of a storage system, the block deduplication module configured to perform a deduplication process based at least in part on the plurality of chunks.

2. The apparatus of claim 1, wherein the file comprises data and a metadata file, the metadata file comprising information describing the structure of the data in the file, the at least one processor further configured to add information to the metadata file indicating that the file has been divided at the at least one location.

3. The apparatus of claim 2, wherein the at least one processing device is configured to hide the metadata file.

4. The apparatus of claim 2, wherein the at least one processing device is configured to access the data in the file randomly based at least in part on the metadata file.

5. The apparatus of claim 1, wherein the at least one location is identified such that the plurality of chunks will be bit aligned.

6. The apparatus of claim 1, wherein the at least one processing device is further configured to add trailing zeroes one or more of the plurality of chunks until the one or more of the plurality of chunks have a size that is a multiple of a block size of the storage system.

7. The apparatus of claim 1, wherein:

determining the type of the file comprises determining that the type is a type that comprises a plurality of sub-files;

determining the structure of the file based at least in part on the determined type of the file comprises determining the structure of the sub-files within the file;

identifying the at least one location in the file based at least in part on the determined structure comprises identifying at least one of a start and an end of at least one of the sub-files; and

dividing the file at the identified at least one location into the plurality of chunks comprises dividing the file at the identified at least one of the start and the end of the at least one of the sub-files.

8. The apparatus of claim 7, wherein:

determining the structure of the file comprises determining that a first of the sub-files and a second of the sub-files together have a size smaller than a block size of the storage system; and

dividing the file at the identified at least one location into the plurality of chunks comprises dividing the file such that the first of the sub-files and the second of the sub-files are included in the same chunk.

9. The apparatus of claim 1, wherein the storage system is a content addressable storage system.

10. The apparatus of claim 1, wherein the processing device is implemented in one of:

a host device configured to communicate over a network with the storage system; and

the storage system.

11. A method comprising:

obtaining a file;

determining a type of the file;

determining a structure of the file based at least in part on the determined type of the file;

identifying at least one location in the file based at least in part on the determined structure;

dividing the file at the identified at least one location into a plurality of chunks; and

providing the plurality of chunks to a block deduplication module of a storage system, the block deduplication module configured to perform a deduplication process based at least in part on the plurality of chunks,

wherein the method is implemented by at least one processing device comprising a processor coupled to a memory.

12. The method of claim 11, wherein the file comprises data and a metadata file, the metadata file comprising information describing the structure of the data in the file, the method further comprising adding information to the metadata file indicating that the file has been divided at the at least one location.

13. The method of claim 12, wherein the at least one processor is configured to hide the metadata file.

14. The method of claim 12, wherein the at least one processor is configured to access the data in the file randomly based at least in part on the metadata file.

15. The method of claim 11, wherein the at least one location is identified such that the plurality of chunks will be bit aligned.

16. The method of claim 11, wherein the at least one processor is further configured to add trailing zeroes one or more of the plurality of chunks until the one or more of the plurality of chunks have a size that is a multiple of a block size of the storage system.

17. The method of claim 11, wherein:

determining the type of the file comprises determining that the type is a type that comprises a plurality of sub-files;

determining the structure of the file based at least in part on the determined type of the file comprises determining the structure of the sub-files within the file;

identifying the at least one location in the file based at least in part on the determined structure comprises identifying at least one of a start and an end of at least one of the sub-files; and

dividing the file at the identified at least one location into the plurality of chunks comprises dividing the file at the identified at least one of the start and the end of the at least one of the sub-files.

18. The method of claim 17, wherein:

determining the structure of the file comprises determining that a first of the sub-files and a second of the sub-files together have a size smaller than a block size of the storage system; and

dividing the file at the identified at least one location into the plurality of chunks comprises dividing the file such that the first of the sub-files and the second of the sub-files are included in the same chunk.

19. The method of claim 17, wherein the at least one processing device is implemented in one of:

a host device configured to communicate over a network with the storage system; and

the storage system.

20. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to obtain a file;

to determine a type of the file;

to determine a structure of the file based at least in part on the determined type of the file;

to identify at least one location in the file based at least in part on the determined structure;

to divide the file at the identified at least one location into a plurality of chunks; and

to provide the plurality of chunks to a block deduplication module of a storage system, the block deduplication module configured to perform a deduplication process based at least in part on the plurality of chunks.