Systems and methods for distributed content storage and management

• US 20040236801A1
• Filed: 05/22/2003
• Published: 11/25/2004
• Est. Priority Date: 05/22/2003
• Status: Active Grant

• Current Assignee: Callahan Cellular LLC (Intellectual Ventures LLC)
• Original Assignee: Einstein's Elephant Incorporated
• Inventors: Borden, Bruce, Brand, Russell
• Granted Patent: US 7,203,711 B2
• Time in Patent Office: Days
• US Class Current: 707/204
• CPC Class Codes: G06F 11/1458 Management of the backup or restore process
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

Systems, methods and computer program products for distributed content storage and management are provided. The distributed content storage and management system includes an indexed archive system; information source agents, information source clients; and a network to couple the information source clients to the indexed archive system. In an alternate embodiment, a legacy back-up system is also used. The index archive system includes an information entryway for gathering file information, a triage engine, an indexing engine, a metadata engine, and a content engine. These engines generate content indexes, metadata and metadata indexes from received files and store them along with the file content into a set of repositories, including an indexing repository, a metadata repository and a content repository. Methods are provided for the efficient retrieval and indexing of files, and for the storage of files, file content, content indexes, metadata and metadata indexes.

Claims

1. A system for archiving and indexing files contained within information source clients, comprising:

(a) a content engine for managing the storage of file content;

(b) an indexing engine for indexing file data content;

(c) a metadata engine for managing metadata characterizing a file; and

(d) a triage engine coupled to and managing said content engine, said indexing engine and said metadata engine.

2. The system of claim 1, further comprising an information entryway coupled to said triage engine, wherein said information entryway receives files from information source clients.

3. The system of claim 1, further comprising an information source modification controller wherein said information source modification controller sends commands to the information source clients.

4. The system of claim 2, further comprising:

(a) an index repository for containing searchable attributes of the file content and/or metadata of files;

(b) a content repository for storing file content; and

(c) a metadata repository for storing metadata, wherein said index repository, said content repository and said metadata repository are coupled to said information entryway.

5. The system of claim 4, further comprising a user interface for managing the system and searching for information about or contained within the indexed files.

6. The system of claim 5, further comprising a search engine coupled to said user interface.

7. The search engine of claim 6, wherein said search engine searches based on indexed file content.

8. The search engine of claim 6, wherein said search engine searches based on metadata.

9. The search engine of claim 6, wherein said search engine searches based on a combination of metadata and indexed file content.

10. The system of claim 1, further comprising a file gathering interface coupled to a legacy back-up system for gathering file information from files being stored by the back-up system.

11. In a network of information source clients wherein a back-up system stores back-up copies of files located on the information source clients to a repository, a system for indexing files being processed by the back-up system, comprising:

(a) an indexing engine for indexing file content;

(b) a metadata engine for managing metadata characterizing a file; and

(c) a triage engine coupled to and managing said file gathering interface, said indexing engine and said metadata engine.

12. The system of claim 11, further comprising a file gathering interface coupled to the back-up system for gathering file information from files being stored by the back-up system;

13. The system of claim 11, further comprising a file administration interface coupled to the back-up system for accessing file administration activity of the back-up system.

14. The system of claim 11, further comprising a file management interface coupled to the back-up system for managing files stored by the back-up system.

15. The system of claim 11, further comprising an information entryway coupled to said triage engine, wherein said information entryway processes information intercepted from files being stored by the back-up system.

16. The system of claim 11, further comprising:

(c) an index repository for containing searchable attributes of the file content and/or metadata of files; and

(d) a metadata repository for storing metadata, wherein said index repository, said content repository and said metadata repository are coupled to said information entryway.

17. The system of claim 16, further comprising a user interface for managing the system and searching for information about or contained within the indexed files.

18. The system of claim 17, further comprising a search engine coupled to said user interface.

19. The search engine of claim 18, wherein said search engine searches based on indexed file content.

20. The search engine of claim 18, wherein said search engine searches based on metadata.

21. The search engine of claim 18, wherein said search engine searches based on a combination of metadata and indexed file content.

22. An information source agent residing within an information source client, comprising:

(a) an information source collection agent for computing, gathering or transporting file data information;

(b) an information source modification agent for making modifications to files located on the information source client; and

(c) an agent controller coupled to said information source collection agent and said information source modification agent for managing their operation and interfacing with external sources.

23. An information source collection agent for computing, gathering or transporting file data information, comprising:

(a) a screening element for accessing whether a file should be transmitted;

(b) an indexing interface for communicating with an indexing system;

(c) an activity monitor for collecting information about file activity; and

(d) a controller coupled to and managing said screening element, said indexing interface and said activity monitor.

24. The information source collection agent of claim 23, wherein said activity monitor collects information about the time a file was deleted.

25. The information source collection agent of claim 23, wherein said activity monitor collects information about renaming of a file.

26. The information source collection agent of claim 23, wherein said activity monitor collects information about file usage.

27. The information source collection agent of claim 23, wherein said activity monitor collects information about intermediate content conditions of files between times when a file is backed-up.

28. The information source collection agent of claim 23, wherein said activity monitor collects information about who has accessed a file.

29. A method for storing file information in an indexed archive system, comprising:

(a) receiving a file that was located on an information source;

(b) generating a file content index for the received file;

(c) extracting the metadata for the received file;

(d) storing the content of the received file;

(e) storing the file content index; and

(f) storing the metadata.

30. The method of claim 29, further comprising requesting that the information source client transmit a file to the indexed archive system.

31. The method of claim 29, further comprising encrypting file content prior to the step of storing file content.

32. The method of claim 29, further comprising sniffing the network to capture all network file access.

33. The method of claim 29, wherein the step of generating metadata is conducted by an information source agent.

34. The method of claim 29, further comprising suppressing the storage of duplicate file contents received from different information source clients.

35. The method of claim 29, further comprising suppressing the storage of duplicate file contents received from the same information source clients.

36. The method of claim 29, wherein said information source client is a desktop computer.

37. The method of claim 29, wherein said information source client is a laptop computer.

38. The method of claim 29, wherein said information source client is a server computer.

39. The method of claim 29, wherein said information source client is a personal digital assistant.

41. The method of claim 29, wherein said information source client is a desktop computer.

42. The method of claim 29, wherein said information source client is a laptop computer.

43. The method of claim 29, wherein said information source client is a server computer.

44. The method of claim 29, wherein said information source client is a personal digital assistant.

40. A method for storing file information in an indexed archive system, comprising:

(a) intercepting file information from a file being stored by a back-up system;

(b) generating a file content index for the received file;

(c) extracting metadata for the received file;

(d) storing a pointer to the location of the intercepted file;

(e) storing the file content index; and

(f) storing the metadata.

45. A method for storing file information in an indexed archive system, comprising:

(a) receiving a file that was located on an information source client;

(b) generating a file content index for the received file; and

(c) comparing the file content index to stored file content indexes.

46. The method of claim 45, wherein the similarity between the file content index and at least one stored file content index exceeds a similarity threshold, further comprising:

(d) comparing differences between the received file and files that have file content indexes that exceed a similarity threshold; and

(e) determining a closest match between the received file and stored files that had file content indexes that exceeded the similarity threshold.

47. The method of claim 46, further comprising:

(f) creating a delta file of the differences between the received file and the closest match file;

(g) updating a file identifier for the closest match file to account for the delta;

(h) storing the received file content; and

(j) storing the file content index.

48. The method of claim 46, further comprising:

(f) creating a delta file of the differences between the received file and the closest match file;

(g) updating a file identifier for the closest match file to account for the delta;

(h) determining whether a storage factor has been met;

(i) when the storage factor has been met storing the delta file, otherwise storing the delta file, the received file, and the file content index.

49. The method of claim 48, wherein step (h) the storage factor is whether the indexed archive system has exceeded a storage threshold.

50. The method of claim 48, wherein step (h) the storage factor is based on file type.

51. The method of claim 48, wherein step (h) the storage factor is based on level of interest of a characteristic of the file.

52. The method of claim 46, further comprising:

(f) determining whether previously received versions of the received file have been previously indexed; and

(g) storing a link to map previously received versions of the received file to the received file.

53. The method of claim 46, further comprising:

(f) storing a link to identify that the received file shares content indices exceeding a similarity threshold for each stored file that has a content index similarity with the received file that exceeds the similarity threshold.

54. The method of claim 45, wherein the similarity between the file content index and at least one stored file content index does not exceed a similarity threshold, further comprising:

(a) storing the received file content; and

(b) storing the file content index for the received file.

55. A method for archiving and indexing files contained within information source clients, comprising:

(a) backing-up files located on information source clients;

(b) indexing files that are backed-up in said step (a) based on content; and

(c) storing file contents backed-up in step (a) and content indexes generated in step (b) to at least one storage device.

56. A system for archiving and indexing files contained within information source clients, comprising:

(a) a back-up system for processing files, file content indexes and file signatures to be stored and indexed;

(b) an indexing-search engine coupled to said back-up system for accessing stored files, file content indexes or file signatures; and

(c) a storage device coupled to said back-up system for the storage of file contents, file content indexes and file signatures.

57. A computer program product comprising a computer useable medium having computer program logic recorded thereon for enabling one or more processors to store file information in an indexed archive system comprising:

(a) receiving means for enabling a processor to receive a file that was located on an information source;

(b) generating means for enabling a processor to generate a file content index for the received file;

(c) extracting means for enabling a processor to extract the metadata for the received file;

(d) storing means for enabling a processor to store the content of the received file;

(e) storing means for enabling a processor to store the file content index; and

(f) storing means for enabling a processor to store the metadata.

58. A computer program product comprising a computer useable medium having computer program logic recorded thereon for enabling one or more processors to store file information in an indexed archive system comprising:

(a) intercepting means for enabling a processor to intercept file information from a file being stored by a back-up system;

(b) generating means for enabling a processor to generate a file content index for the received file;

(c) extracting means for enabling a processor to extract metadata for the received file;

(d) storing means for enabling a processor to store a pointer to the location of the intercepted file;

(e) storing means for enabling a processor to store the file content index; and

(f) storing means for enabling a processor to store the metadata.

59. A computer program product comprising a computer useable medium having computer program logic recorded thereon for enabling one or more processors to store file information in an indexed archive system comprising:

(a) receiving means for enabling a processor to receive a file that was located on an information source client;

(b) generating means for enabling a processor to generate a file content index for the received file; and

(c) comparing means for enabling a processor to compare the file content index to stored file content indexes.