Methods and systems for object level de-duplication for solid state devices

• US 10,162,553 B2
• Filed: 11/23/2011
• Issued: 12/25/2018
• Est. Priority Date: 11/24/2010
• Status: Active Grant

• Current Assignee: JP Morgan Chase Bank, N.A. (JP Morgan Chase & Co)
• Original Assignee: Western Digital Technologies Incorporated (Western Digital Corporation)
• Inventors: Sadri, Mohammad R., Choudhuri, Siddharth
• Primary Examiner(s): Rones, Charles
• Assistant Examiner(s): Chang, Tian-Pong
• Application Number:
  US13/303,870
• Publication Number:
  US 20120137054A1
• Time in Patent Office: 2,589 Days
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays

Abstract

In one aspect, the present disclosure relates to a method of de-duplicating data in a solid state storage device. The method can include receiving a block of data to be written to a solid state storage device, wherein the block of data comprises header portion and a payload, wherein the header portion comprises context information; and determining whether the payload should be de-duplicated prior to storage, based on the context information stored within the header portion; if the payload is determined to be de-duplicated, de-duplicating the payload; and storing the de-duplicated payload to the solid state storage device.

Claims

1. A method for providing object level de-duplication for solid state storage devices (SSDs), comprising:

receiving a block of data at a solid state storage device (SSD), wherein the block of data comprises a header portion and a payload, and wherein the header portion comprises context information;

determining, by the SSD, that the payload should be de-duplicated prior to storage based solely on the context information stored within the header portion without analyzing the payload;

compressing the payload such that the compressed payload includes random data;

building a hash, the hash consisting of a subset of the random data less than an entirety of the compressed payload;

de-duplicating the compressed payload using the hash; and

storing the de-duplicated payload to the solid state storage device.

2. The method of claim 1, wherein the context information is provided from hierarchical layers above the SSD.

3. The method of claim 2, wherein the context information is indicative of whether the payload comprises random data.

4. The method claim 1, wherein payload data comprises a plurality of sub-objects.

5. The method of claim 4 comprising de-duplicating based on a correlation between sub-objects of other payloads.

6. The method of claim 1, comprising before storing, separating out the header from payload and storing the header and the payload in separate locations on the SSD.

7. The method of claim 1, further comprising determining that the payload is not to be de-duplicated when the cost of improperly determining to de-duplicate the payload is high.

8. The method of claim 1, further comprising deriving the context information from the header portion by analyzing the header portion in view of a known structure of the header portion.

9. The method of claim 3, further comprising de-duplicating the payload only when the payload is non-random.

10. The method of claim 1, wherein the context information comprises a sensitivity level associated with the payload, indicative of a cost of improperly determining to de-duplicate the payload.

11. The method of claim 1, wherein the context information is uniquely associated with the payload.

12. A solid state device (SSD) providing object level de-duplication, comprising:

a host interface for communicating to a host device;

a flash memory for storing data; and

a de-duplication engine comprising logic stored in a computer readable medium, the logic configured to;

receive a block of data, wherein the block of data comprises a header portion and a payload, and wherein the header portion comprises context information;

determine the payload should be de-duplicated prior to storage based solely on the context information stored within the header portion without analyzing the payload;

compress the payload such that the compressed payload includes random data;

build a hash, the hash consisting of a subset of the random data less than an entirety of the compressed payload;

de-duplicate the compressed payload using the hash; and

store the de-duplicated payload to the flash memory.

13. The solid state device of claim 12, wherein the context information is provided from hierarchical layers above the SSD.

14. The solid state device of claim 12, wherein payload data comprises a plurality of sub-objects.

15. The solid state device of claim 12, wherein the logic is further configured to determine that the payload is not to be de-duplicated when the cost of improperly determining to de-duplicate the payload is high.

16. A method for providing object level de-duplication for solid state storage devices (SSDs), comprising:

receiving, at a solid state storage device (SSD), a first block of data to be written to the SSD, wherein the first block of data comprises first header information and a first object;

determining, at the SSD based solely on the first header information without analyzing the first object, the first object should be de-duplicated prior to storage;

compressing the first object such that the first object includes random data;

building a hash, the hash consisting of a subset of the random data less than an entirety of the compressed first object;

de-duplicating, at the SSD, the compressed first object using the hash; and

storing the de-duplicated first object to the SSD.

17. The method of claim 16, wherein the first header information is provided by an application layer that had previously accessed the first block of data.

18. The method of claim 16, wherein determining, based on the first header information, the first object should be de-duplicated prior to storage, comprises:

deriving, at the SSD, context information based on the first header information by analyzing the header portion in view of a known structure of the header portion; and

determining, at the SSD, based on the context information, whether the first object should be de-duplicated prior to storage.

19. The method of claim 16, wherein the first block of data further comprises second header information and a second object, wherein the method further comprises determining, at the SSD, based on the first header information and the second header information, whether the second object should be de-duplicated prior to storage.

20. The method of claim 16, further comprising:

receiving, at the SSD, a second block of data to be written to the SSD, wherein the second block of data comprises second header information and a second object;

determining, at the SSD, based on the first header information and the second header information, whether the second object should be de-duplicated prior to storage.