Efficient data storage using two level delta resemblance

• US 8,145,863 B2
• Filed: 04/13/2011
• Issued: 03/27/2012
• Est. Priority Date: 04/11/2006
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Zhu, Ming Benjamin
• Primary Examiner(s): Dudek, Jr., Edward
• Assistant Examiner(s): Verderamo, III, Ralph A
• Application Number:
  US13/086,332
• Publication Number:
  US 20110191560A1
• Time in Patent Office: 349 Days
• Field of Search: None
• US Class Current: 711/162
• CPC Class Codes: G06F 11/1451 by selection of backup contents

Abstract

Storage using resemblance of data segments is disclosed. It is determined that a new segment resembles a second prior stored segment wherein the second prior stored segment is represented as a first stored delta and a first prior stored segment. A second delta between the new segment and the prior stored segment is determined. A representation of the new segment based at least in part on the second delta is stored.

Claims

1. A system for storage using resemblance of data segments comprising:

a processor configure to;

break up a new input data stream into a plurality of data segments, wherein a new segment comprises one of the plurality of data segments, and wherein the breaking up of the new input data stream is based at least in part on a hash of content of the data stream;

determine that a new segment resembles a second prior stored segment wherein the second prior stored segment is represented as a first stored delta and a first prior stored segment, wherein resemblance is based at least in part on a summary feature set, wherein the summary feature set is determined by (a) selecting m subsegments of a first segment;

(b) selecting n different functions, and wherein n is greater than 1; and

(c) for function i, wherein i is a value from 1 to n, computing m values f_i(subsegment_j) wherein j is a value from 1 to m; and

(d) selecting a subset of values from the f_i(subsegment_j) values, wherein the f_i(subsegment_j) values encompass all of the values as a result of applying the n different functions to the m subsegments and

determine a second delta between the new segment and the second prior stored segment; and

store a representation of the new segment based at least in part on the second delta; and

a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions.

2. A system as in claim 1, wherein determining resemblance comprises comparing a summary feature set of the new segment with a summary feature set of the second prior stored segment.

3. A system as in claim 1, wherein the processor is further configured to store the new segment in the event that the new segment does not resemble the second prior stored segment.

4. A system as in claim 1, wherein storing the representation of the new segment comprises storing the second delta, an indication of the first stored delta, and an indication of the first prior stored segment.

5. A system as in claim 1, wherein storing the representation of the new segment in the event that the second delta is larger than a predetermined threshold comprises storing the new segment.

6. A system as in claim 1, wherein storing the representation of the new segment in the event that the second delta is larger than a predetermined threshold and the new segment resembles the first prior stored segment comprises storing a third delta and an indication of the first prior stored segment.

7. A system as in claim 6, wherein the third delta comprises a difference between the new segment and the first prior stored segment.

8. A system as in claim 6, wherein determining resemblance comprises comparing a summary feature set of the new segment with a summary feature set of the first prior stored segment.

9. A system as in claim 6, wherein storing the representation of the new segment comprises storing the third delta combined with a fifth delta.

10. A system as in claim 6, wherein storing the representation of the new segment comprises locally compressing the third delta.

11. A system as in claim 1, wherein storing the representation of the new segment comprises storing the second delta combined with a fourth delta.

12. A system as in claim 1, wherein storing the representation of the new segment comprises locally compressing the new segment.

13. A system as in claim 1, wherein storing the representation of the new segment comprises locally compressing the second delta.

14. A system as in claim 1, wherein an identifier is computed for the new segment.

15. A system as in claim 1, wherein an identifier is computed for the new segment and comprises a fingerprint.

16. A system as in claim 1, wherein an identifier is computed for the new segment and comprises a serially generated number.

17. A system as in claim 1, wherein storing the representation of the new segment comprises storing the representation of the new segment in a backup storage system.

18. A system as in claim 1, wherein there is no loss of information in storing the new segment.

19. The system as in claim 1, wherein the n functions operating on the m subsegments comprises more than one of the following:

a sum of the bytes in the subsegment, a set of linear functions, a hash function, SHA-1 (Secure Hash Algorithm

     1), MD5 (Message-Digest algorithm

     5), RIPEMD-160 (RACE Integrity Primitives Evaluation Message Digest 160-bit version), and a Rabin hash.

20. A method for storage using resemblance of data segments, comprising:

breaking up a new input data stream into a plurality of data segments, wherein a new segment comprises one of the plurality of data segments, and wherein the breaking up of the new input data stream is based at least in part on a hash of content of the data stream;

determining that a new segment resembles a second prior stored segment wherein the second prior stored segment is represented as a first stored delta and a first prior stored segment, wherein resemblance is based at least in part on a summary feature set, wherein the summary feature set is determined by (a) selecting m subsegments of a first segment;

(b) selecting n different functions, and wherein n is greater than 1; and

(c) for function i, wherein i is a value from 1 to n, computing m values f_i(subsegment_j) wherein j is a value from 1 to m; and

(d) selecting a subset of values from the f_i(subsegment_j) values, wherein the f_i(subsegment_j) values encompass all of the values as a result of applying the n different functions to the m subsegments; and

determining a second delta between the new segment and the second prior stored segment; and

storing a representation of the new segment based at least in part on the second delta.

21. A computer program product for storage using resemblance of data segments, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions for:

breaking up a new input data stream into a plurality of data segments, wherein a new segment comprises one of the plurality of data segments, and wherein the breaking up of the new input data stream is based at least in part on a hash of content of the data stream;

determining that a new segment resembles a second prior stored segment wherein the second prior stored segment is represented as a first stored delta and a first prior stored segment, wherein resemblance is based at least in part on a summary feature set, wherein the summary feature set is determined by (a) selecting m subsegments of a first segment;

(b) selecting n different functions, and wherein n is greater than 1; and

(c) for function i, wherein i is a value from 1 to n, computing m values f_i(subsegment_j) wherein j is a value from 1 to m; and

(d) selecting a subset of values from the f_i(subsegment_j) values, wherein the f_i(subsegment_j) values encompass all of the values from applying the n different functions to the m subsegments; and

determining a second delta between the new segment and the second prior stored segment; and

storing a representation of the new segment based at least in part on the second delta.