Method and system for endurance enhancing, deferred deduplication with hardware-hash-enabled storage device

• US 10,437,784 B2
• Filed: 01/29/2016
• Issued: 10/08/2019
• Est. Priority Date: 01/30/2015
• Status: Active Grant

• Current Assignee: SK Hynix Inc.
• Original Assignee: SK Hynix Inc.
• Inventors: Chai, Zhenchuan, Harijono, Indra G., Han, Hwansoo
• Primary Examiner(s): Beausoliel, Jr., Robert W
• Assistant Examiner(s): Rayyan, Susan F
• Application Number:
  US15/011,017
• Publication Number:
  US 20160224589A1
• Time in Patent Office: 1,348 Days
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A storage system may include at least one storage device and a server. The storage device may store an incoming data, calculate a hash value for the incoming data, and store the hash value as meta data. The server may provide the incoming data to the storage device, read the meta data from the storage device, determine whether the meta data is duplicate in the table, and perform a deduplication process regarding the incoming data based on the determination result.

Claims

1. A system comprising:

at least one storage device configured to store an incoming data, calculate a hash value for the incoming data, and store the hash value as meta data; and

a server configured to provide the incoming data to the storage device, wherein, when a later-scheduled post-deduplication operation is to be performed, as part of the later-scheduled post-deduplication operation, the server reads the meta data from the storage device, determines whether the meta data is duplicate in a mapping table, and performs a deduplication process regarding the incoming data based on the determination result.

2. The system of claim 1, wherein the server is configured to delete the duplicated copy of the incoming data in the storage device, if the meta data is duplicate.

3. The system of claim 1, wherein the storage device includes:

a hash engine configured to calculate the hash value for the incoming data; and

a storage configured to store the incoming data and the meta data.

4. The system of claim 3, wherein the storage device includes a solid state drive (SSD).

5. The system of claim 1, wherein the storage device includes:

a primary storage device configured to store the incoming data; and

a secondary storage device including;

a hash engine configured to calculate the hash value for the incoming data; and

a storage configured to store the meta data.

6. The system of claim 5, wherein the secondary storage device includes a non-volatile random access memory (NVRAM) card configured to be mounted on the server.

7. A method comprising:

providing, by a server, an incoming data to at least one storage device;

storing, by the storage device, the incoming data;

calculating, by the storage device, a hash value for the incoming data;

storing, by the storage device, the hash value as meta data;

reading, by the server, when a later-scheduled post-deduplication operation is to be performed, as part of the later-scheduled post-deduplication operation, the meta data from the storage device;

determining, by the server, whether the meta data is duplicate in a mapping table maintained by the server; and

performing, by the server, a deduplication process regarding the incoming data based on the determination result.

8. The method of claim 7, wherein performing of the deduplication process comprises deleting the duplicated copy of the incoming data in the storage device, if the meta data is duplicate.

9. The method of claim 7, wherein the storage device includes a solid state drive (SSD).

10. The method of claim 7, wherein the storage device includes:

a primary storage device configured to store the incoming data; and

a secondary storage device including;

a hash engine configured to calculate the hash value for the incoming data; and

a storage configured to store the meta data.

11. The method of claim 10, wherein the secondary storage device includes a non-volatile random access memory (NVRAM) card configured to be mounted on the server.

12. A storage server comprising:

a receiver and interface assembly configured to;

receive an incoming data; and

provide the incoming data to at least one storage device external to the storage server, and

a processor configured to;

read meta data as a hash value for the incoming data from the storage device when a later-scheduled post-deduplication operation is to be performed, as part of the later-scheduled post-deduplication operation;

determine whether the meta data is duplicate in a mapping table; and

perform a deduplication process regarding the incoming data in the storage device based on the determination result.

13. The server of claim 12, wherein the processor is configured to delete the duplicated copy of the incoming data in the storage device, if the meta data is duplicate.

14. The server of claim 12, wherein the storage device includes a solid state drive (SSD).

15. A storage server comprising:

an internal storage device mounted on the server, configured to store an incoming data, calculate a hash value for the incoming data, and store the hash value as meta data; and

a processing unit configured to read the meta data from the internal storage device when a later-scheduled post-deduplication operation is to be performed, as part of the later-scheduled post-deduplication operation, determining whether the meta data is duplicate in a mapping table, and performing a deduplication process regarding the incoming data based on the determination result.

16. The server of claim 15, wherein the processing unit is configured to delete the duplicated copy of the incoming data in the internal storage device, if the meta data is duplicate.

17. The server of claim 15, wherein the processing unit is configured to provide the incoming data to at least one external storage device, if the meta data is non-duplicate.

18. The server of claim 15, wherein the internal storage device includes:

a hash engine configured to calculate the hash value for the incoming data; and

a storage configured to store the incoming data and the meta data.

19. The server of claim 18, wherein the internal storage device includes a non-volatile random access memory (NVRAM) card configured to be mounted on the server.