Tape drive library integrated memory deduplication

• US 10,289,335 B2
• Filed: 09/12/2017
• Issued: 05/14/2019
• Est. Priority Date: 09/12/2017
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated (Kyndryl Holdings, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Asmussen, Ole, Beiderbeck, Robert, Rueger, Erik, Schafer, Markus
• Primary Examiner(s): Tabone, Jr., John T
• Application Number:
  US15/702,158
• Publication Number:
  US 20190079690A1
• Time in Patent Office: 609 Days
• Field of Search: 714747, 714718, 714771, 714769
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0686 Libraries, e.g. tape libraries, jukebox

Abstract

A method and system for improving tape drive memory storage is provided. The method includes receiving, by a storage tape drive hardware device, a data stream. Duplicate data chunks of the data stream are identified and deleted such that a group of data chunks remain in a non-volatile memory device (NVS2) of the storage tape drive. The group of data chunks is written to a data storage tape cartridge. Pointers are generated and stored within the data storage tape cartridge. The pointers are associated with a location within an NVS1 for storing de-duplication hashes identifying each data chunk of the group of data chunks stored within data storage tape cartridge. The de-duplication hashes are written by from the NVS1 to a manager non-volatile memory device (MNVS) of a storage tape drive hardware library system. The MNVS is updated.

Claims

1. A tape drive memory storage improvement method comprising:

receiving, by a processor of a storage tape drive hardware device of a storage tape drive hardware library system comprising a plurality of storage tape drive hardware devices, a data stream for storage, wherein said storage tape drive hardware device internally comprises a deduplication software engine, a non-volatile memory device (NVS1), a non-volatile memory device (NVS2), and a first data storage tape cartridge;

identifying, by said processor within said NVS2, duplicate data chunks of a plurality of adjacent variable length data chunks of said data stream, wherein said duplicate data chunks comprise duplicated data with respect to a first group of data chunks of said plurality of adjacent variable length data chunks;

deleting, by said processor from said NVS2, said duplicate data chunks such that said first group of data chunks remain within said NVS2;

writing, by said processor to a first data storage tape cartridge of said storage tape drive hardware device, said first group of data chunks;

generating, by said processor, pointers associated with a location within said NVS1 storing de-duplication hashes identifying each data chunk of said first group of data chunks stored within said first data storage tape cartridge;

storing, by said processor, said pointers within said first data storage tape cartridge;

writing, by said processor from said NVS1 to a manager non-volatile memory device (MNVS) of said storage tape drive hardware library system, said de-duplication hashes; and

updating, by said processor, said MNVS by combining said de-duplication hashes with a plurality of additional de-duplication hashes identifying a plurality of commonly used de-duplicated data chunks stored within said plurality of storage tape drive hardware devices.

2. The method of claim 1, wherein said first data storage tape cartridge is removed from said storage tape drive hardware device, wherein a second data storage tape cartridge is placed within a second storage tape drive hardware device of said plurality of storage tape drive hardware devices, and wherein said method further comprises:

writing, from said MNVS to an additional NVS1 of said second storage tape drive hardware device, said de-duplication hashes combined with said plurality of additional de-duplication hashes;

receiving an additional data stream for storage;

dividing said additional data stream into a second plurality of adjacent variable length data chunks;

storing, within an additional NVS2 of said second storage tape drive hardware device, said second plurality of adjacent variable length data chunks;

de-duplicating, based on said de-duplication hashes combined with said plurality of additional de-duplication hashes, said second plurality of adjacent variable length data chunks such that a second group of de-duplicated data chunks remain within said additional NVS2;

writing, by said processor to said second data storage tape cartridge of said second storage tape drive hardware device, said second group of de-duplicated data chunks;

generating, by said processor, additional pointers associated with a location within said additional NVS1 storing said de-duplication hashes combined with said plurality of additional de-duplication hashes; and

storing, by said processor, said pointers within said second data storage tape cartridge.

3. The method of claim 2, wherein said storage tape drive hardware library system further comprises a communication component, and wherein said method further comprises:

synchronizing with all de-duplication hashes within said MNVS, via said communication component, all additional de-duplication hashes associated with all additional de-duplicated data chunks processed by said plurality of storage tape drive hardware devices and said storage tape drive hardware library system such that said MNVS is updated with currently updated de-duplication hashes associated with currently updated de-duplicated data chunks.

4. The method of claim 2, further comprising:

detecting corrupted data chunks stored within a storage tape drive hardware device of said plurality of storage tape drive hardware devices;

retrieving, from said MNVS, pointers associated with functional data chunks of at least one additional NVS2 of said plurality of storage tape drive hardware devices comprising data storage tape cartridges including said functional data chunks;

replacing said corrupted data chunks with said functional data chunks.

5. The method of claim 1, further comprising:

providing an Ethernet connection between said MNVS and said plurality of storage tape drive hardware devices.

6. The method of claim 1, wherein said MNVS comprises a memory device selected from the group consisting of an integrated circuit based memory device and a removable flash memory device.

7. The method of claim 1, wherein said MNVS comprises a hash database.

8. The method of claim 1, further comprising:

encrypting, by said processor, all hashes within said MNVS.

9. The method of claim 1, further comprising:

providing at least one support service for at least one of creating, integrating, hosting, maintaining, and deploying computer-readable code in the storage tape drive hardware device, said code being executed by the processor to implement;

said receiving, said identifying, said deleting, said writing said first group of data chunks, said generating said pointers, said storing said pointers, said writing said de-duplication hashes, and said updating.

10. A computer program product, comprising a computer readable hardware storage device storing a computer readable program code, said computer readable program code comprising an algorithm that when executed by a processor of a storage tape drive hardware device implements a tape drive memory storage improvement method, said method comprising:

receiving, by said processor, a data stream for storage, wherein said storage tape drive hardware device is comprised by a storage tape drive hardware library system comprising a plurality of storage tape drive hardware devices, and wherein said storage tape drive hardware device internally comprises a deduplication software engine, a non-volatile memory device (NVS1), a non-volatile memory device (NVS2), and a first data storage tape cartridge;

identifying, by said processor within said NVS2, duplicate data chunks of a plurality of adjacent variable length data chunks of said data stream, wherein said duplicate data chunks comprise duplicated data with respect to a first group of data chunks of said plurality of adjacent variable length data chunks;

deleting, by said processor from said NVS2, said duplicate data chunks such that said first group of data chunks remain within said NVS2;

writing, by said processor to a first data storage tape cartridge of said storage tape drive hardware device, said first group of data chunks;

generating, by said processor, pointers associated with a location within said NVS1 storing de-duplication hashes identifying each data chunk of said first group of data chunks stored within said first data storage tape cartridge;

storing, by said processor, said pointers within said first data storage tape cartridge;

writing, by said processor from said NVS1 to a manager non-volatile memory device (MNVS) of said storage tape drive hardware library system, said de-duplication hashes; and

updating, by said processor, said MNVS by combining said de-duplication hashes with a plurality of additional de-duplication hashes identifying a plurality of commonly used de-duplicated data chunks stored within said plurality of storage tape drive hardware devices.

11. The computer program product of claim 10, wherein said first data storage tape cartridge is removed from said storage tape drive hardware device, wherein a second data storage tape cartridge is placed within a second storage tape drive hardware device of said plurality of storage tape drive hardware devices, and wherein said method further comprises:

writing, from said MNVS to an additional NVS1 of said second storage tape drive hardware device, said de-duplication hashes combined with said plurality of additional de-duplication hashes;

receiving an additional data stream for storage;

dividing said additional data stream into a second plurality of adjacent variable length data chunks;

storing, within an additional NVS2 of said second storage tape drive hardware device, said second plurality of adjacent variable length data chunks;

de-duplicating, based on said de-duplication hashes combined with said plurality of additional de-duplication hashes, said second plurality of adjacent variable length data chunks such that a second group of de-duplicated data chunks remain within said additional NVS2;

writing, by said processor to said second data storage tape cartridge of said second storage tape drive hardware device, said second group of de-duplicated data chunks;

generating, by said processor, additional pointers associated with a location within said additional NVS1 storing said de-duplication hashes combined with said plurality of additional de-duplication hashes; and

storing, by said processor, said pointers within said second data storage tape cartridge.

12. The computer program product of claim 11, wherein said storage tape drive hardware library system further comprises a communication component, and wherein said method further comprises:

synchronizing with all de-duplication hashes within said MNVS, via said communication component, all additional de-duplication hashes associated with all additional de-duplicated data chunks processed by said plurality of storage tape drive hardware devices and said storage tape drive hardware library system such that said MNVS is updated with currently updated de-duplication hashes associated with currently updated de-duplicated data chunks.

13. The computer program product of claim 11, wherein said method further comprises:

detecting corrupted data chunks stored within a storage tape drive hardware device of said plurality of storage tape drive hardware devices;

retrieving, from said MNVS, pointers associated with functional data chunks of at least one additional NVS2 of said plurality of storage tape drive hardware devices comprising data storage tape cartridges including said functional data chunks;

replacing said corrupted data chunks with said functional data chunks.

14. The computer program product of claim 10, wherein said method further comprises:

providing an Ethernet connection between said MNVS and said plurality of storage tape drive hardware devices.

15. The computer program product of claim 10, wherein said MNVS comprises a memory device selected from the group consisting of an integrated circuit based memory device and a removable flash memory device.

16. The computer program product of claim 10, wherein said MNVS comprises a hash database.

17. The computer program product of claim 10, further comprising:

encrypting, by said processor, all hashes within said MNVS.

18. A storage tape drive hardware device comprising a processor coupled to a computer-readable memory unit, said memory unit comprising instructions that when executed by the processor implements a tape drive memory storage improvement method comprising:

receiving, by said processor, a data stream for storage, wherein said storage tape drive hardware device is comprised by a storage tape drive hardware library system comprising a plurality of storage tape drive hardware devices, and wherein said storage tape drive hardware device internally comprises a deduplication software engine, a non-volatile memory device (NVS1), a non-volatile memory device (NVS2), and a first data storage tape cartridge;

identifying, by said processor within said NVS2, duplicate data chunks of a plurality of adjacent variable length data chunks of said data stream, wherein said duplicate data chunks comprise duplicated data with respect to a first group of data chunks of said plurality of adjacent variable length data chunks;

deleting, by said processor from said NVS2, said duplicate data chunks such that said first group of data chunks remain within said NVS2;

writing, by said processor to a first data storage tape cartridge of said storage tape drive hardware device, said first group of data chunks;

generating, by said processor, pointers associated with a location within said NVS1 storing de-duplication hashes identifying each data chunk of said first group of data chunks stored within said first data storage tape cartridge;

storing, by said processor, said pointers within said first data storage tape cartridge;

writing, by said processor from said NVS1 to a manager non-volatile memory device (MNVS) of said storage tape drive hardware library system, said de-duplication hashes; and

updating, by said processor, said MNVS by combining said de-duplication hashes with a plurality of additional de-duplication hashes identifying a plurality of commonly used de-duplicated data chunks stored within said plurality of storage tape drive hardware devices.

19. The storage tape drive hardware device of claim 18, wherein said first data storage tape cartridge is removed from said storage tape drive hardware device, wherein a second data storage tape cartridge is placed within a second storage tape drive hardware device of said plurality of storage tape drive hardware devices, and wherein said method further comprises:

writing, from said MNVS to an additional NVS1 of said second storage tape drive hardware device, said de-duplication hashes combined with said plurality of additional de-duplication hashes;

receiving an additional data stream for storage;

dividing said additional data stream into a second plurality of adjacent variable length data chunks;

storing, within an additional NVS2 of said second storage tape drive hardware device, said second plurality of adjacent variable length data chunks;

de-duplicating, based on said de-duplication hashes combined with said plurality of additional de-duplication hashes, said second plurality of adjacent variable length data chunks such that a second group of de-duplicated data chunks remain within said additional NVS2;

writing, by said processor to said second data storage tape cartridge of said second storage tape drive hardware device, said second group of de-duplicated data chunks;

generating, by said processor, additional pointers associated with a location within said additional NVS1 storing said de-duplication hashes combined with said plurality of additional de-duplication hashes; and

storing, by said processor, said pointers within said second data storage tape cartridge.

20. The storage tape drive hardware device of claim 19, wherein said storage tape drive hardware library system further comprises a communication component, and wherein said method further comprises:

synchronizing with all de-duplication hashes within said MNVS, via said communication component, all additional de-duplication hashes associated with all additional de-duplicated data chunks processed by said plurality of storage tape drive hardware devices and said storage tape drive hardware library system such that said MNVS is updated with currently updated de-duplication hashes associated with currently updated de-duplicated data chunks.