Distributed Catalog, Data Store, and Indexing

• US 20110225141A1
• Filed: 03/14/2011
• Published: 09/15/2011
• Est. Priority Date: 03/12/2010
• Status: Active Grant

• Current Assignee: Malikie Innovations Limited (Key Patent Innovations Limited)
• Original Assignee: Copiun Incorporated (Blackberry Limited)
• Inventors: Jain, Sanjay, Chaudhry, Puneesh
• Granted Patent: US 9,135,264 B2
• Time in Patent Office: Days
• US Class Current: 707/709
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files

Abstract

This disclosure relates to a system and method for distributed catalog processing, data caching, and indexing to create an efficient, scalable, secure, high availability, disaster recovery enabled backup and storage system.

Claims

1. A method comprising:

comparing a local catalog on a first computer system to a production file system to identify a first file, the first file having metadata and existing within the production file system;

identifying a set of storage blocks to represent the first file;

transferring the first file to a local cache on the first computer system,

wherein the first file is represented in the local cache as the set of storage blocks, and

wherein the set of storage blocks is a subset of a larger set of local cache blocks that represent other files within the production file system, and

wherein each of the local cache blocks is capable of being stored both within the local cache and on a storage server on a second computer system;

creating a delta entry in the local catalog on the first computer system, wherein the delta entry includes metadata relating to the first file and associated with an indication of the time the delta entry was created;

transferring the first file to the storage server by transferring the set of storage blocks to the storage server; and

updating a remote catalog on the second computer system to reflect the metadata in the delta entry.

2. The method of claim 1, further comprising:

determining if the local cache has reached a size limit;

if the local cache has reached the size limit, identifying, within the set of local cache blocks in the local cache, a candidate block that had previously been backed up successfully to the storage server, wherein identifying the candidate block comprises comparing a timestamp of the candidate block with a time of a last successful storage of the local cache to the storage server; and

deleting the candidate block to reduce the size of the local cache.

3. The method of claim 2, further comprising repopulating the local cache with storage blocks that had previously been deleted from the local cache.

4. The method of claim 1, further comprising:

building on the first computer system a searchable index of files, the files including the first file, wherein a file pointer is associated with the first file, and wherein the file pointer is also associated with the identity of the set of storage blocks that represent the first file;

identifying, using the index, the file pointer to the first file;

identifying, using the file pointer, the set of storage blocks that represent the first file, wherein the set of storage blocks include a subset of local blocks that are located on the first computer system, and a subset of remote blocks that are not located on the first computer system;

transferring to the first computer system the subset of remote blocks; and

reconstructing the first file from a combination of the local and remote blocks.

5. The method of claim 4, wherein building the searchable index of files includes crawling for file contents using the file pointer.

6. The method of claim 1, wherein the first file was modified after a first point in time, and wherein only those entries in the local catalog that were entered after the first point in time are sent to the second computer system.

7. The method of claim 1 wherein the first file is identified at least in part because the first file was deleted from the production file system.

8. The method of claim 1 wherein the first file is identified at least in part because the first file was added to or modified within the production file system.

9. The method of claim 1, wherein the local catalog is configured to include entries corresponding to multiple storage clients.

10. The method of claim 1, wherein transferring the first file to the storage server includes:

determining if the first file can be sent to the storage server on the second computer system; and

transferring the first file to the storage server and updating the remote catalog at a point in time when the first file can be sent to the storage server.

11. The method of claim 1, wherein the steps of comparing a local catalog to a production file system, identifying a set of storage blocks, transferring to a local cache, creating a delta entry in the local catalog, transferring to a storage server, and updating a remote catalog are performed by a third and a fourth computer system such that the third and fourth computer systems use the same storage server and remote catalog as the first computer system uses.

12. Logic encoded in one or more non-transient media that includes code for execution and when executed by a processor is operable to perform operations comprising:

comparing a local catalog on a first computer system to a production file system to identify a first file, the first file having metadata and existing within the production file system;

identifying a set of storage blocks to represent the first file;

transferring the first file to a local cache on the first computer system,

wherein the first file is represented in the local cache as the set of storage blocks, and

wherein the set of storage blocks is a subset of a larger set of local cache blocks that represent other files within the production file system, and

wherein each of the local cache blocks is capable of being stored both within the local cache and on a storage server on a second computer system;

creating a delta entry in the local catalog on the first computer system, wherein the delta entry includes metadata relating to the first file and associated with an indication of the time the delta entry was created;

transferring the first file to the storage server by transferring the set of storage blocks to the storage server; and

updating a remote catalog on the second computer system to reflect the metadata in the delta entry.

13. The logic of claim 12, further comprising:

determining if the local cache has reached a size limit;

if the local cache has reached the size limit, identifying, within the set of local cache blocks in the local cache, a candidate block that had previously been backed up successfully to the storage server, wherein identifying the candidate block comprises comparing a timestamp of the candidate block with a time of a last successful transfer of the local cache to the storage server; and

deleting the candidate block to reduce the size of the local cache.

14. The logic of claim 13, further comprising repopulating the local cache with storage blocks that had previously been deleted from the local cache.

15. The logic of claim 12, further comprising:

building on the first computer system a searchable index of files, the files including the first file, wherein a file pointer is associated with the first file, and wherein the file pointer is also associated with the identity of the set of storage blocks that represent the first file;

identifying, using the index, the file pointer to the first file;

identifying, using the file pointer, the set of storage blocks that represent the first file, wherein the set of storage blocks include a subset of local blocks that are located on the first computer system, and a subset of remote blocks that are not located on the first computer system;

transferring to the first computer system the subset of remote blocks; and

reconstructing the first file from a combination of the local and remote blocks.

16. The logic of claim 15, wherein building the searchable index of files includes crawling for file contents using the file pointer.

17. The logic of claim 12, wherein the first file was modified after a first point in time, and wherein only those entries in the local catalog that were entered after the first point in time are sent to the second computer system.

18. The logic of claim 12, wherein the first file is identified at least in part because the first file was deleted from the production file system.

19. The logic of claim 12, wherein the first file is identified at least in part because the first file was added to or modified within the production file system.

20. A system comprising:

a local catalog on a first computer system;

a local cache on the first computer system;

a production file system;

a remote catalog on a second computer system;

a storage server on the second computer system;

an agent running on the first computer;

a memory on the first computer system capable of storing data; and

a processor on the first computer system configured for;

comparing a local catalog on a first computer system to a production file system to identify a first file, the first file having metadata and existing within the production file system;

identifying a set of storage blocks to represent the first file;

transferring the first file to a local cache on the first computer system,

wherein the first file is represented in the local cache as the set of storage blocks, and

wherein the set of storage blocks is a subset of a larger set of local cache blocks that represent other files within the production file system, and

wherein each of the local cache blocks is capable of being stored both within the local cache and on a storage server on a second computer system;

creating a delta entry in the local catalog on the first computer system, wherein the delta entry includes metadata relating to the first file and associated with an indication of the time the delta entry was created;

transferring the first file to the storage server by transferring the set of storage blocks to the storage server; and

updating a remote catalog on the second computer system to reflect the metadata in the delta entry.

21. The system of claim 20, further comprising:

determining if the local cache has reached a size limit;

if the local cache has reached the size limit, identifying, within the set of local cache blocks in the local cache, a candidate block that had previously been backed up successfully to the storage server, wherein identifying the candidate block comprises comparing a timestamp of the candidate block with a time of a last successful transfer of the local cache to the storage server; and

deleting the candidate block to reduce the size of the local cache.

22. The system of claim 21, further comprising repopulating the local cache with storage blocks that had previously been deleted from the local cache.

23. The system of claim 20, further comprising:

building on the first computer system a searchable index of files, the files including the first file, wherein a file pointer is associated with the first file, and wherein the file pointer is also associated with the identity of the set of storage blocks that represent the first file;

identifying, using the index, the file pointer to the first file;

identifying, using the file pointer, the set of storage blocks that represent the first file, wherein the set of storage blocks include a subset of local blocks that are located on the first computer system, and a subset of remote blocks that are not located on the first computer system;

transferring to the first computer system the subset of remote blocks; and

reconstructing the first file from a combination of the local and remote blocks.

24. The system of claim 23, wherein building the searchable index of files includes crawling for file contents using the file pointer.

25. The system of claim 20, wherein the first file was modified after a first point in time, and wherein only those entries in the local catalog that were entered after the first point in time are sent to the second computer system.

26. The system of claim 20, wherein the first file is identified at least in part because the first file was deleted from the production file system.

27. The system of claim 20, wherein the first file is identified at least in part because the first file was added to or modified within the production file system.