DATA COMPRESSION AND STORAGE TECHNIQUES

• US 20080034268A1
• Filed: 04/09/2007
• Published: 02/07/2008
• Est. Priority Date: 04/07/2006
• Status: Active Grant

• Current Assignee: Data Storage Group, Inc.
• Original Assignee: Data Storage Group, Inc.
• Inventors: Dodd, Brian, Moore, Michael
• Granted Patent: US 7,860,843 B2
• Time in Patent Office: Days
• US Class Current: 714/755
• CPC Class Codes: H03M 7/30 Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction

Abstract

Provided are systems and methods for use in data archiving. In one arrangement, compression techniques are provided wherein an earlier version of a data set (e.g., file folder, etc) is utilized as a dictionary of a compression engine to compress a subsequent version of the data set. This compression identifies changes between data sets and allows for storing these differences without duplicating many common portions of the data sets. For a given version of a data set, new information is stored along with metadata used to reconstruct the version from each individual segment saved at different points in time. In this regard, the earlier data set and one or more references to stored segments of a subsequent data set may be utilized to reconstruct the subsequent data set.

Claims

1. A method for use in archiving/storing data, comprising:

receiving a first data set;

compressing the first data set using a dictionary-based compression engine, wherein a first compressed file representing the first data set is generated;

storing the first compressed file; and

using the first compressed file to identify changes in a subsequent data set associated with the first data set.

2. The method of claim 1, wherein using the first compressed file to identify changes comprises:

conditioning a dictionary-based compression engine with the first compressed file to define a conditioned compression engine; and

compressing the subsequent data set using the conditioned compression engine, wherein a second compressed file representing the subsequent data set is generated.

3. The method of claim 2, wherein the second compression file comprises:

changes between the first data set and the subsequent data set; and

one or more references to the first compressed file.

4. The method of claim 3, further comprising:

storing the second compressed file.

5. The method of claim 1, generating at least a first hash associated with the first data set.

6. The method of claim 5, wherein identifying changes further comprises:

comparing at least one hash of the subsequent data set with a corresponding hash of the first data set.

7. The method of claim 6, wherein a difference is identified between the corresponding hashes, further comprising:

conditioning a dictionary-based compression engine with the first compressed file to define a conditioned compression engine; and

compressing the subsequent data set using the conditioned compression engine, wherein a second compressed file representing the subsequent data set is generated.

8. The method of claim 5, wherein generating at least a first hash comprises:

generating a first hash associated with meta data of the first data set.

9. The method of claim 8, further comprising:

generating a second hash associated with content of the first data set.

10. The method of claim 1, further comprising:

buffering the subsequent data set in portions for the compression engine; and

substantially matching the buffered portions of the subsequent first data set with like sized portions of the compressed file after the compressed file is uncompressed.

11. A method for compressing data comprising:

receiving a file;

determining that a previous version of the file has been previously stored;

compressing the file using compression dictionary terms generated from the previous version of the file, wherein a compressed file is generated, and

storing the compressed file.

12. The method of claim 11, wherein the previously stored version of the file is a previously compressed file.

13. The method of claim 12, further comprising:

preloading a compression engine with the previously compressed file;

buffering the received file in portions for the compression engine; and

substantially matching the buffered portions of the received file with like sized portions of the previously compressed file after the previously compressed file is uncompressed.

14. The method of claim 11, wherein the determining step is accomplished by hashing a version reference for the file and failing to match a resulting version reference hash with a previously stored version reference hash;

and hashing a secondary reference for the file and successfully matching a resulting secondary reference hash with a previously stored secondary reference hash.

15. The method of claim 11, further comprising generating and storing a version reference associated with the file.

16. The method of claim 15, further comprising generating and storing a secondary reference associated with the file.

17. The method of claim 16, wherein the version reference and the secondary reference are both hashes.

18. A method for compressing data comprising:

storing an initial data set;

preloading a compression engine with the initial data set, wherein the initial data set defines a dictionary for the compression engine;

compressing a subsequent data set associated with the initial data set using the preloaded compression engine, wherein compressing the subsequent data set generates a compressed file; and

storing the compressed file.

19. The method of claim 18, wherein the compressed files comprises changes between the initial data set and the subsequent data set and one or more references to the first data set.

20. The method of claim 18, further comprising:

identifying the association between the initial data set and the subsequent data set based on at least one meta-data value.

21-55. -55. (canceled)