Information source agent systems and methods for distributed data storage and management using content signatures

• US 8,392,705 B2
• Filed: 03/24/2010
• Issued: 03/05/2013
• Est. Priority Date: 05/22/2003
• Status: Active Grant

• Current Assignee: Callahan Cellular LLC (Intellectual Ventures LLC)
• Original Assignee: Carmenso Data Limited Liability Company (Intellectual Ventures LLC)
• Inventors: Borden, Bruce, Brand, Russell
• Primary Examiner(s): ARMOUCHE, HADI S
• Application Number:
  US12/730,741
• Publication Number:
  US 20100180128A1
• Time in Patent Office: 1,077 Days
• Field of Search: 713/165, 713/176, 707/E17.009, 707/E17.041, 707/E17.005, 707/803, 707/812, 707662-664, 707/692, 707697-699, 707/747, 707/758, 707/780, 705/51, 709/224
• US Class Current: 713/165
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/1756 based on delta files
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1873 Versioning file systems, temporal file systems, e.g. file system supporting different historic versions of files
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload

Abstract

Information source agent systems and methods for distributed content storage and management using content signatures that use file identicality properties are provided. A data management system is provided that includes a content engine for managing the storage of file content, a content signature generator that generates a unique content signature for a file processed by the content engine, a content signature comparator that compares content signatures and a content signature repository that stores content signatures. Information source agents are provided that include content signature generators and content signature comparators. Methods are provided for the efficient management of files using content signatures that take advantage of file identicality properties. Content signature application modules and registries exist within information source clients and centralized servers to support the content signature methods.

Claims

1. A method comprising:

generating a file content index for a file received at a computing device from an information source client;

comparing the file content index to stored file content indices to determine a similarity between the file content index and the stored file content indices and to determine whether the similarity exceeds a similarity threshold;

comparing, if the similarity exceeds the similarity threshold, differences between the received file and files that have the file content indices that exceed the similarity threshold;

determining, from among the files that have the file content indices that exceed the similarity threshold, a closest match file to the received file; and

creating a delta file of the differences between the received file and the closest match file.

2. The method of claim 1, further comprising:

updating a file identifier for the closest match file to account for the differences;

storing the content of the received file; and

storing the file content index,

wherein the file identifier comprises a unique value generated from the received file.

3. The method of claim 1, further comprising:

storing the delta file if a storage factor is above a threshold value.

4. The method of claim 1, further comprising:

storing at least one of the delta file, the received tile, and the file content index if a storage factor is below a threshold value.

5. The method of claim 4, wherein the threshold is a storage threshold of an indexed archive system.

6. The method of claim 4, wherein the threshold is based on file type.

7. The method of claim 4, wherein the threshold is based on level of interest of a characteristic of the received file.

8. The method of claim 1, further comprising:

determining whether previously received versions of the received file have been previously indexed; and

storing a link to map the previously received versions of the received file to the received file.

9. The method of claim 1, further comprising:

storing a link indicating the received file shares content indices exceeding a similarity threshold for at least one stored file.

10. An non-transitory computer readable medium having stored thereon in digital form computer-executable instructions that, in response to execution by a computing device, cause the computing device to perform operations for storing file information in an indexed archive system, the operations comprising:

generating a file content index for a file received at the computing device from an information source client;

comparing the file content index to stored file content indices to determine a similarity between the file content index and the stored file content indices and to determine whether the similarity exceeds a similarity threshold;

comparing, if the similarity exceeds the similarity threshold, differences between the received file and files that have the file content indices that exceed the similarity threshold;

determining, from among the files that have the file content indices that exceed the similarity threshold, a closest match file to the received file; and

creating a delta file of the differences between the received file and the closest match file.

11. The non-transitory computer readable medium of claim 10, the operations further comprising:

updating a file identifier for the closest match file to account for the differences;

storing the content of the received file; and

storing the file content index,

wherein the file identifier comprises a unique value generated from the received file.

12. The non-transitory computer readable medium of claim 10, the operations further comprising:

storing the delta file if a storage factor is above a threshold value.

13. The non-transitory computer readable medium of claim 10, the operations further comprising:

storing at least one of the delta file, the received file, and the file content index if a storage factor is below a threshold value.

14. The non-transitory computer readable medium of claim 13, wherein the threshold is a storage threshold of the indexed archive system.

15. The non-transitory computer readable medium of claim 13, wherein the threshold is based on file type.

16. The non-transitory computer readable medium of claim 13, wherein the threshold is based on level of interest of a characteristic of the received file.

17. The non-transitory computer readable medium of claim 10, the operations further comprising:

determining whether previously received versions of the received file have been previously indexed; and

storing a link to map the previously received versions of the received file to the received file.

18. The non-transitory computer readable medium of claim 10, the operations further comprising:

storing a link indicating the received file shares content indices exceeding a similarity threshold for at least one stored file.