Efficient data storage system

• US 8,275,955 B2
• Filed: 06/21/2010
• Issued: 09/25/2012
• Est. Priority Date: 12/20/2002
• Status: Expired due to Term

• Current Assignee: EMC Corporation (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Zhu, Ming Benjamin, Patterson, R. Hugo, Li, Kai
• Primary Examiner(s): Thai, Tuan V.
• Application Number:
  US12/819,356
• Publication Number:
  US 20100257315A1
• Time in Patent Office: 827 Days
• Field of Search: 711/100, 711/118, 711/154, 711/156, 711/161, 711/162
• US Class Current: 711/154
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0683 Plurality of storage devices
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A system and method are disclosed for providing efficient data storage. A plurality of data segments is received in a data stream. The system determines whether a data segment has been stored previously in a low latency memory. In the event that the data segment is determined to have been stored previously, an identifier for the previously stored data segment is returned.

Claims

1. A system for storing data comprising:

an interface for receiving a data stream comprising a first plurality of data segments;

a low latency memory for storing a second plurality of data segments;

a processor configured to;

generate a segment information for a segment of the first plurality of data segments, wherein the segment information includes a digital signature generated based at least in part on a content of the segment of the first plurality of data segments, wherein the digital signature is a Rabin fingerprint;

determine that one of the first plurality of data segments has been stored previously in the low latency memory as one of the second plurality of data segments;

in the event that the one of the first plurality of data segments is determined to have been stored previously, returning an identifier of one of the second plurality of data segments;

in the event that one of the first plurality of data segments is not determined to have been stored, confirming whether the data segment has been stored; and

in the event that one of the first plurality of data segments is confirmed to not have been stored, storing the data segment.

2. A system as in claim 1, wherein the determination can positively confirm that the one of the first data segments has been previously stored, but cannot positively confirm that the one of the first data segments has not been previously stored.

3. A system as in claim 1, wherein the identifier is derived from a content of the one of the second plurality of data segment.

4. A system as in claim 1, wherein the identifier is not derived from a content of the one of the second plurality of data segment.

5. A system as in claim 1, wherein the identifier is not found in the low latency memory.

6. A system as in claim 1, wherein the identifier is used to reconstruct the data stream.

7. A system as in claim 1, wherein the processor is further configured to discard the one of the first plurality of data segments.

8. A system as in claim 1, wherein the determination does not require accessing a high latency memory.

9. A system as in claim 1, wherein the determination does not require accessing a disk.

10. A system as in claim 1, wherein the processor is further configured to generate segment information for the first plurality of data segments.

11. A system as in claim 1, wherein the processor is further configured to generate segment information for the first plurality of data segments;

wherein the segment information includes a short identifier.

12. A system as in claim 1, wherein the processor is further configured to generate segment information for the first plurality of data segments;

wherein the segment information includes a digital signature and a short identifier.

13. A system as in claim 1, wherein the processor is further configured to generate segment information for the first plurality of data segments;

wherein the segment information includes a digital signature;

wherein the digital signature is a result of a hash function.

14. A system as in claim 1, wherein determining whether the one of the first plurality of data segments has been stored previously includes checking a cache.

15. A system as in claim 1, wherein the processor is further configured to generate segment information for the data segments, wherein determining whether the one of the first plurality of data segments has been stored previously comprises looking up a segment information in a cache to determine whether the one of the first plurality of data segments has been previously stored.

16. A computer program product for storing data, the computer program product being embodied in a computer readable medium and comprising computer instructions for:

receiving a data stream comprising a first plurality of data segments;

generating a segment information for the first plurality of data segments, wherein the segment information includes a digital signature generated based at least in part on a content of the segment of the first plurality of data segments, wherein the digital signature is a Rabin fingerprint;

determining, using a processor, that one of the first plurality of data segments has been stored previously in a low latency memory as one of a second plurality of data segments;

in the event that the one of the first plurality of data segments is determined to have been stored previously, returning an identifier for the one of the second plurality of data segments;

in the event that one of the first plurality of data segments is not determined to have been stored, confirming whether the data segment has been stored; and

in the event that one of the first plurality of data segments is confirmed to not have been stored, storing the data segment.

17. A method for storing data comprising:

receiving a data stream comprising a first plurality of data segments;

generating a segment information for the first plurality of data segments, wherein the segment information includes a digital signature generated based at least in part on a content of the segment of the first plurality of data segments, wherein the digital signature is a Rabin fingerprint;

determining, using a processor, that one of the first plurality of data segments has been stored previously in a low latency memory as one of a second plurality of data segments;

in the event that the one of the first plurality of data segments is determined to have been stored previously, returning an identifier for the one of the second plurality of data segments;

in the event that one of the first plurality of data segments is not determined to have been stored, confirming whether the data segment has been stored; and

in the event that one of the first plurality of data segments is confirmed to not have been stored, storing the data segment.

18. A method as in claim 17, wherein the determination can positively confirm that the one of the first data segments has been previously stored, but cannot positively confirm that the one of the first data segments has not been previously stored.

19. A method as in claim 17, wherein the identifier is derived from a content of the one of the second plurality of data segment.

20. A method as in claim 17, wherein the identifier is not derived from a content of the one of the second plurality of data segment.

21. A method as in claim 17, wherein the identifier is not found in the low latency memory.

22. A method as in claim 17, wherein the identifier is used to reconstruct the data stream.

23. A method as in claim 17, wherein the determination does not require accessing a disk.

24. A method as in claim 17, wherein the processor is further configured to generate segment information for the first plurality of data segments.

25. A method as in claim 17, further comprising generating segment information for the first plurality of data segments;

wherein the segment information includes a short identifier.

26. A method as in claim 17, further comprising generating segment information for the first plurality of data segments;

wherein the segment information includes a digital signature and a short identifier.

27. A method as in claim 17, further comprising generating segment information for the first plurality of data segments;

wherein the segment information includes a digital signature;

wherein the digital signature is a result of a hash function.

28. A method as in claim 17, wherein determining whether the one of the first plurality of data segments has been stored previously includes checking a cache.

29. A method as in claim 17, further comprising generating segment information for the data segments, wherein determining whether the one of the first plurality of data segments has been stored previously comprises looking up a segment information in a cache to determine whether the one of the first plurality of data segments has been previously stored.