Tape drive memory deduplication

• US 10,372,681 B2
• Filed: 09/12/2017
• Issued: 08/06/2019
• Est. Priority Date: 09/12/2017
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated (Kyndryl Holdings, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Asmussen, Ole, Beiderbeck, Robert, Rueger, Erik, Schafer, Markus
• Primary Examiner(s): Verbrugge, Kevin
• Application Number:
  US15/701,994
• Publication Number:
  US 20190079947A1
• Time in Patent Office: 693 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0682 Tape device
• CPC Class Codes: H03M 7/3091 Data deduplication
• CPC Class Codes: H03M 7/3095 using variable length segments

Abstract

A method and system for improving tape drive memory storage is provided. The method includes receiving, by a storage tape drive, a data stream for storage. The data stream is passed through a non-volatile memory device (NVS2) of the storage tape drive. The data stream is divided into adjacent variable length data chunks and a chunk list file including similarity identifiers for each of the adjacent variable length data chunks is generated and stored within a (non-volatile memory device) NVS1. Duplicate data including duplicated data with respect to a group of data chunks of the adjacent variable length data chunks is identified and deleted from the NVS2 of the storage tape drive such that the group of data chunks remains within NVS2. The group of data chunks is written to a data storage tape cartridge. Pointers identifying each data chunk and an associated storage position are generated and stored.

Claims

1. A tape drive memory storage improvement method comprising:

receiving, by a processor of a storage tape drive hardware device, a data stream, wherein said storage tape drive hardware device internally comprises a deduplication software engine, a first non-volatile memory device (NVS1), a second non-volatile memory device (NVS2), and a first data storage tape cartridge;

passing, by said processor through said NVS2, said data stream;

dividing, by said processor executing said deduplication software engine within said NVS2, said data stream into a plurality of adjacent variable length data chunks;

generating, by said processor, a chunk list file comprising similarity identifiers associated with each of said plurality of adjacent variable length data chunks;

storing, by said processor within said NVS1, said chunk list file;

identifying, by said processor, duplicate data chunks of said plurality of adjacent variable length data chunks, wherein said duplicate data chunks comprise duplicated data with respect to a first group of data chunks of said plurality of adjacent variable length data chunks;

deleting, by said processor from said NVS2, said duplicate data chunks such that said first group of data chunks remain within said NVS2;

writing, by said processor from said NVS2 to said first data storage tape cartridge, said first group of data chunks for storage;

generating, by said processor, pointers identifying each data chunk of said first group of data chunks and an associated storage position, within said first data storage tape cartridge, for each said data chunk of said first group of data chunks;

storing, by said processor, said pointers within said chunk list file located within said NVS1; and

writing, by said processor from said NVS1 to said first data storage tape cartridge, said chunk list file comprising said pointers for storage.

2. The method of claim 1, wherein said first data storage tape cartridge is removed from said storage tape drive hardware device, and wherein said method further comprises:

deleting, by said processor from said NVS1, said chunk list file comprising said pointers; and

deleting, by said processor from said NVS2, said first group of data chunks.

3. The method of claim 2, wherein a second data storage tape cartridge is placed within said storage tape drive hardware device, and wherein said method further comprises:

writing, by said processor from said second data storage tape cartridge to said NVS2, a second group of de-duplicated data chunks;

writing, by said processor from said second data storage tape cartridge to said NVS1, an additional chunk list file comprising pointers identifying each data chunk of said second group of de-duplicated data chunks and an associated storage position;

identifying, by said processor based on analysis of said additional chunk list file, necessary duplicate data chunks comprising duplicated data with respect to said second group of de-duplicated data chunks; and

adding, by said processor to said second group of de-duplicated data chunks, said necessary duplicate data chunks such that said second group of de-duplicated data chunks in combination with said necessary duplicate data chunks comprise a complete data file for execution.

4. The method of claim 3, wherein said second data storage tape cartridge is removed from said storage tape drive hardware device, wherein a third data storage tape cartridge is placed within said storage tape drive hardware device, and wherein said method further comprises:

receiving, by said processor, a second data stream for storage;

executing, by said processor based on said second group of de-duplicated data chunks of said NVS2 and said additional chunk list file of said NVS1, a de-duplication process with respect to said second data stream resulting in a third group of de-duplicated data chunks for storage; and

writing, by said processor to said third data storage tape cartridge, said third group of de-duplicated data chunks for storage.

5. The method of claim 1, wherein said identifying is executed until a specified data storage size threshold with respect to said data stream has been exceeded.

6. The method of claim 1, further comprising:

partitioning, by said processor said NVS2 such that said data stream is stored within a first partition of said NVS2 and said first group of data chunks is stored within a second partition of said NVS2.

7. The method of claim 1, wherein said NVS1 comprises a first integrated circuit based memory device, and wherein said NVS2 comprises a second integrated circuit based memory device.

8. The method of claim 1, further comprising:

providing at least one support service for at least one of creating, integrating, hosting, maintaining, and deploying computer-readable code in the storage tape drive hardware device, said code being executed by the processor to implement;

said receiving, said storing said data stream, said dividing, said generating said chunk list file, said storing said chunk list file, said identifying, said deleting, said writing said first group of data chunks, said generating said pointers, said storing said pointers, and said writing said chunk list file.

9. A computer program product, comprising a computer readable hardware storage device storing a computer readable program code, said computer readable program code comprising an algorithm that when executed by a processor of a storage tape drive hardware device implements a tape drive memory storage improvement method, said method comprising:

receiving, by said processor, a data stream for storage, wherein said storage tape drive hardware device internally comprises a deduplication software engine, a first non-volatile memory device (NVS1), a second non-volatile memory device (NVS2), and a first data storage tape cartridge;

passing, by said processor through said NVS2, said data stream;

dividing, by said processor executing said deduplication software engine within said NVS2, said data stream into a plurality of adjacent variable length data chunks;

generating, by said processor, a chunk list file comprising similarity identifiers associated with each of said plurality of adjacent variable length data chunks;

storing, by said processor within said NVS1, said chunk list file;

identifying, by said processor, duplicate data chunks of said plurality of adjacent variable length data chunks, wherein said duplicate data chunks comprise duplicated data with respect to a first group of data chunks of said plurality of adjacent variable length data chunks;

deleting, by said processor from said NVS2, said duplicate data chunks such that said first group of data chunks remain within said NVS2;

writing, by said processor from said NVS2 to said first data storage tape cartridge, said first group of data chunks for storage;

generating, by said processor, pointers identifying each data chunk of said first group of data chunks and an associated storage position, within said first data storage tape cartridge, for each said data chunk of said first group of data chunks;

storing, by said processor, said pointers within said chunk list file located within said NVS1; and

writing, by said processor from said NVS1 to said first data storage tape cartridge, said chunk list file comprising said pointers for storage.

10. The computer program product of claim 9, wherein said first data storage tape cartridge is removed from said storage tape drive hardware device, and wherein said method further comprises:

deleting, by said processor from said NVS1, said chunk list file comprising said pointers; and

deleting, by said processor from said NVS2, said first group of data chunks.

11. The computer program product of claim 10, wherein a second data storage tape cartridge is placed within said storage tape drive hardware device, and wherein said method further comprises:

writing, by said processor from said second data storage tape cartridge to said NVS2, a second group of de-duplicated data chunks;

writing, by said processor from said second data storage tape cartridge to said NVS1, an additional chunk list file comprising pointers identifying each data chunk of said second group of de-duplicated data chunks and an associated storage position; and

identifying, by said processor based on analysis of said additional chunk list file, necessary duplicate data chunks comprising duplicated data with respect to said second group of de-duplicated data chunks; and

adding, by said processor to said second group of de-duplicated data chunks, said necessary duplicate data chunks such that said second group of de-duplicated data chunks in combination with said necessary duplicate data chunks comprise a complete data file for execution.

12. The computer program product of claim 11, wherein said second data storage tape cartridge is removed from said storage tape drive hardware device, wherein a third data storage tape cartridge is placed within said storage tape drive hardware device, and wherein said method further comprises:

receiving, by said processor, a second data stream for storage;

executing, by said processor based on said second group of de-duplicated data chunks of said NVS2 and said additional chunk list file of said NVS1, a de-duplication process with respect to said second data stream resulting in a third group of de-duplicated data chunks for storage; and

writing, by said processor to said third data storage tape cartridge, said third group of de-duplicated data chunks for storage.

13. The computer program product of claim 9, wherein said identifying is executed until a specified data storage size threshold with respect to said data stream has been exceeded.

14. The computer program product of claim 9, wherein said method further comprises:

partitioning, by said processor said NVS2 such that said data stream is stored within a first partition of said NVS2 and said first group of data chunks is stored within a second partition of said NVS2.

15. The computer program product of claim 9, wherein said NVS1 comprises a first integrated circuit based memory device, and wherein said NVS2 comprises a second integrated circuit based memory device.

16. A storage tape drive hardware device comprising a processor coupled to a computer-readable memory unit, said memory unit comprising instructions that when executed by the processor implements a tape drive memory storage improvement method comprising:

receiving, by said processor, a data stream for storage, wherein said storage tape drive hardware device internally comprises a deduplication software engine, a first non-volatile memory device (NVS1), a second non-volatile memory device (NVS2), and a first data storage tape cartridge;

passing, by said processor through said NVS2, said data stream;

dividing, by said processor executing said deduplication software engine within said NVS2, said data stream into a plurality of adjacent variable length data chunks;

generating, by said processor, a chunk list file comprising similarity identifiers associated with each of said plurality of adjacent variable length data chunks;

storing, by said processor within said NVS1, said chunk list file;

identifying, by said processor, duplicate data chunks of said plurality of adjacent variable length data chunks, wherein said duplicate data chunks comprise duplicated data with respect to a first group of data chunks of said plurality of adjacent variable length data chunks;

deleting, by said processor from said NVS2, said duplicate data chunks such that said first group of data chunks remain within said NVS2;

writing, by said processor from said NVS2 to said first data storage tape cartridge, said first group of data chunks for storage;

generating, by said processor, pointers identifying each data chunk of said first group of data chunks and an associated storage position, within said first data storage tape cartridge, for each said data chunk of said first group of data chunks;

storing, by said processor, said pointers within said chunk list file located within said NVS1; and

writing, by said processor from said NVS1 to said first data storage tape cartridge, said chunk list file comprising said pointers for storage.

17. The storage tape drive hardware device of claim 16, wherein said first data storage tape cartridge is removed from said storage tape drive hardware device, and wherein said method further comprises:

deleting, by said processor from said NVS1, said chunk list file comprising said pointers; and

deleting, by said processor from said NVS2, said first group of data chunks.

18. The storage tape drive hardware device of claim 17, wherein a second data storage tape cartridge is placed within said storage tape drive hardware device, and wherein said method further comprises:

writing, by said processor from said second data storage tape cartridge to said NVS2, a second group of de-duplicated data chunks;

writing, by said processor from said second data storage tape cartridge to said NVS1, an additional chunk list file comprising pointers identifying each data chunk of said second group of de-duplicated data chunks and an associated storage position; and

identifying, by said processor based on analysis of said additional chunk list file, necessary duplicate data chunks comprising duplicated data with respect to said second group of de-duplicated data chunks; and

adding, by said processor to said second group of de-duplicated data chunks, said necessary duplicate data chunks such that said second group of de-duplicated data chunks in combination with said necessary duplicate data chunks comprise a complete data file for execution.

19. The storage tape drive hardware device of claim 18, wherein said second data storage tape cartridge is removed from said storage tape drive hardware device, wherein a third data storage tape cartridge is placed within said storage tape drive hardware device, and wherein said method further comprises:

receiving, by said processor, a second data stream for storage;

executing, by said processor based on said second group of de-duplicated data chunks of said NVS2 and said additional chunk list file of said NVS1, a de-duplication process with respect to said second data stream resulting in a third group of de-duplicated data chunks for storage; and

writing, by said processor to said third data storage tape cartridge, said third group of de-duplicated data chunks for storage.

20. The storage tape drive hardware device of claim 16, wherein said identifying is executed until a specified data storage size threshold with respect to said data stream has been exceeded.

21. A tape drive memory storage improvement method comprising:

receiving, by a processor of a storage tape drive hardware device, a data file for storage, wherein said storage tape drive hardware device internally comprises a deduplication software engine, a first non-volatile memory device (NVS1), a second non-volatile memory device (NVS2), and a first data storage tape cartridge;

dividing, by said processor executing said deduplication software engine, said data file into a plurality of adjacent variable length data chunks;

identifying, by said processor executing said deduplication software engine, duplicate data chunks of said plurality of adjacent variable length data chunks, wherein said duplicate data chunks comprise duplicated data with respect to a first group of data chunks of said plurality of adjacent variable length data chunks;

storing, by said processor within a first database within said NVS2, said first group of data chunks;

generating, by said processor, pointers identifying each data chunk of said first group of data chunks and an associated storage position, within said first database of said NVS2, for each said data chunk of said first group of data chunks;

storing, by said processor within a second database within said NVS1, said pointers;

first writing, by said processor from said NVS2 to said first data storage tape cartridge, said first group of data chunks for storage; and

second writing, by said processor from said NVS1 to said first data storage tape cartridge, said pointers.

22. The method of claim 21, further comprising:

before said first writing, encrypting by said processor, said first group of data chunks; and

before said second writing, encrypting by said processor, said pointers.

23. The method of claim 21, wherein each of said NVS1 and said NVS2 comprises a memory device selected from the group consisting of an integrated circuit based memory device and a removable flash memory device.

24. A computer program product, comprising a computer readable hardware storage device storing a computer readable program code, said computer readable program code comprising an algorithm that when executed by a processor of a storage tape drive hardware device implements a tape drive memory storage improvement method, said method comprising:

receiving, by said processor, a data file for storage, wherein said storage tape drive hardware device internally comprises a deduplication software engine, a first non-volatile memory device (NVS1), a second non-volatile memory device (NVS2), and a first data storage tape cartridge;

dividing, by said processor executing said deduplication software engine, said data file into a plurality of adjacent variable length data chunks;

identifying, by said processor executing said deduplication software engine, duplicate data chunks of said plurality of adjacent variable length data chunks, wherein said duplicate data chunks comprise duplicated data with respect to a first group of data chunks of said plurality of adjacent variable length data chunks;

storing, by said processor within a first database within said NVS2, said first group of data chunks;

generating, by said processor, pointers identifying each data chunk of said first group of data chunks and an associated storage position, within said first database of said NVS2, for each said data chunk of said first group of data chunks;

storing, by said processor within a second database within said NVS1, said pointers;

first writing, by said processor from said NVS2 to said first data storage tape cartridge, said first group of data chunks for storage; and

second writing, by said processor from said NVS1 to said first data storage tape cartridge, said pointers.

25. The computer program product of claim 24, wherein each of said NVS1 and said NVS2 comprises a memory device selected from the group consisting of an integrated circuit based memory device and a removable flash memory device.