EFFICIENT DATA STORAGE SYSTEM

• US 20100257315A1
• Filed: 06/21/2010
• Published: 10/07/2010
• Est. Priority Date: 12/20/2002
• Status: Active Grant

• Current Assignee: EMC Corporation (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Zhu, Ming Benjamin, Li, Kai, Patterson, R. Hugo
• Granted Patent: US 8,275,955 B2
• Time in Patent Office: Days
• US Class Current: 711/118
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 3/0608 Saving storage space on storage systems
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0683 Plurality of storage devices
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD

Abstract

A system and method are disclosed for providing efficient data storage. A plurality of data segments is received in a data stream. The system determines whether a data segment has been stored previously in a low latency memory. In the event that the data segment is determined to have been stored previously, an identifier for the previously stored data segment is returned.

Claims

1. A system for storing data comprising:

an interface for receiving a data stream comprising a first plurality of data segments;

a low latency memory for storing a second plurality of data segments;

a processor configured to

determine that one of the first plurality of data segments has been stored previously in the low latency memory as one of the second plurality of data segments;

in the event that the one of the first plurality of data segments is determined to have been stored previously, return an identifier for the one of the second plurality of data segments.

2. A system as in claim 1, wherein the determination can positively confirm that the one of is the first data segments has been previously stored, but cannot positively confirm that the one of the first data segments has not been previously stored.

3. A system as in claim 1, wherein the identifier is derived from a content of the one of the second plurality of data segment.

4. A system as in claim 1, wherein the identifier is not derived from a content of the one of the second plurality of data segment.

5. A system as in claim 1, wherein the identifier is not found in the low latency memory.

6. A system as in claim 1, wherein the identifier is used to reconstruct the data stream.

7. A system as in claim 1, where in the processor is further configured to:

in the event that the one of the first plurality of data segments is not determined to have been stored, confirm whether the one of the first plurality of data segments has been stored; and

in the event that the data segment is confirmed to have been stored, store the data segment.

8. A system as in claim 1, wherein the processor is further configured to discard the one of the first plurality of data segments.

9. A system as in claim 1, wherein the determination does not require accessing a high latency memory.

10. A system as in claim 1, wherein the determination does not require accessing a disk.

11. A system as in claim 1, wherein the processor is further configured to generate segment information for the first plurality of data segments.

12. A system as in claim 1, wherein the processor is further configured to generate segment information for the first plurality of data segments;

wherein the segment information includes a digital signature.

13. A system as in claim 1, wherein the processor is further configured to generate segment information for the first plurality of data segments;

wherein the segment information includes a short identifier that is not likely to be unique.

14. A system as in claim 1, wherein the processor is further configured to generate segment information for the first plurality of data segments;

wherein the segment information includes a digital signature and a short identifier that is not likely to be unique.

15. A system as in claim 1, wherein the processor is further configured to generate segment information for the first plurality of data segments;

wherein the segment information includes a digital signature;

wherein the digital signature is a result of a hash function.

16. A system as in claim 1, wherein the processor is further configured to generate segment information for the data segment;

wherein the segment information includes a digital signature;

wherein the digital signature is a Rabin fingerprint.

17. A system as in claim 1, wherein determining whether the one of the first plurality of data segments has been stored previously includes checking a cache.

18. A system as in claim 1, wherein the processor is further configured to generate segment information for the data segments, wherein determining whether the one of the first plurality of data segments has been stored previously comprises looking up a segment information in a cache to determine whether the one of the first plurality of data segments has been previously stored.

19. A computer program product for storing data, the computer program product being embodied in a computer readable medium and comprising computer instructions for:

receiving a data stream comprising a first plurality of data segments;

determining that one of the first plurality of data segments has been stored previously in a low latency memory as one of a second plurality of data segments;

in the event that the one of the first plurality of data segments is determined to have been stored previously, returning an identifier for the one of the second plurality of data segments.

20. A method for storing data comprising:

receiving a data stream comprising a first plurality of data segments;

determining that one of the first plurality of data segments has been stored previously in a low latency memory as one of a second plurality of data segments;

in the event that the one of the first plurality of data segments is determined to have been stored previously, returning an identifier for the one of the second plurality of data segments.