Content-based segmentation scheme for data compression in storage and transmission including hierarchical segment representation

• US 8,508,389 B2
• Filed: 11/04/2010
• Issued: 08/13/2013
• Est. Priority Date: 10/30/2002
• Status: Expired due to Term

• Current Assignee: Riverbed Technology, LLC (Riverbed Technology Incorporated)
• Original Assignee: Riverbed Technology Incorporated
• Inventors: McCanne, Steven, Demmer, Michael J.
• Primary Examiner(s): NGUYEN, LINH V
• Application Number:
  US12/940,017
• Publication Number:
  US 20110267207A1
• Time in Patent Office: 1,013 Days
• Field of Search: 341/50, 341/51, 341/65, 341/67, 341/107, 709/246, 709/247
• US Class Current: 341/51
• CPC Class Codes: H03M 7/30 Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

In a coding system, input data within a system is encoded. The input data might include sequences of symbols that repeat in the input data or occur in other input data encoded in the system. The encoding includes determining a target segment size, determining a window size, identifying a fingerprint within a window of symbols at an offset in the input data, determining whether the offset is to be designated as a cut point and segmenting the input data as indicated by the set of cut points. For each segment so identified, the encoder determines whether the segment is to be a referenced segment or an unreferenced segment, replacing the segment data of each referenced segment with a reference label and storing a reference binding in a persistent segment store for each referenced segment, if needed. Hierarchically, the process can be repeated by grouping references into groups, replacing the grouped references with a group label, storing a binding between the grouped references and group label, if one is not already present, and repeating the process. The number of levels of hierarchy can be fixed in advanced or it can be determined from the content encoded.

Claims

1. A method for saving a file to a storage system comprising:

receiving a file or a portion of a file;

compressing the file or portion thereof by;

at least partially segmenting the file or portion thereof into segments based on content of the file;

determining for each segment, whether the segment is a recurring segment;

responsive to determining that a respective segment is a recurring segment, replacing the segment with a corresponding reference;

responsive to determining that a respective segment is not a recurring segment, creating a reference for the segment and replacing the segment with the created reference; and

saving the file or portion thereof to the storage system.

2. A method for saving a file to a storage system comprising compressing a file by:

segmenting at least portions of the file into segments based on content of the file;

determining for each segment, whether the segment is a recurring segment;

responsive to determining that a respective segment is a recurring segment, replacing the segment with a corresponding reference; and

responsive to determining that a respective segment is not a recurring segment, creating a reference for the segment and replacing the segment with the created reference; and

storing the compressed file in the storage system.

3. The method of claim 2, wherein a size of the segments is not fixed.

4. The method of claim 2, wherein segmenting the file comprises generating a fingerprint of a group of data symbols.

5. The method of claim 4, wherein segmenting the file comprises determining a segmentation boundary based on a respective fingerprint.

6. The method of claim 4, wherein generating the fingerprint comprises generating a hash based on the group of data symbols.

7. The method of claim 2, wherein compressing the file further comprises compressing references hierarchically.

8. The method of claim 7, wherein compressing the file references hierarchically comprises assigning a hash to a sequence of hashes.

9. The method of claim 7, wherein compressing the file references hierarchically comprises:

segmenting a sequence of references into groups;

determining, for each group, whether the group is a recurring group;

responsive to determining that a respective group is a recurring group, replacing the group with a corresponding reference; and

responsive to determining that a respective group is not a recurring group, creating a reference for the group and replacing the group with the created reference.

10. The method of claim 7, wherein the compressed file includes references associated with different hierarchy levels.

11. The method of claim 2, further comprising storing segment-to-reference binding information in a persistent storage.

12. The method of claim 2, further comprising allowing an average segment size to be set.