System and method for policy based synchronization of remote and local file systems

• US 10,380,076 B2
• Filed: 07/21/2015
• Issued: 08/13/2019
• Est. Priority Date: 07/21/2014
• Status: Active Grant

• Current Assignee: Egnyte Incorporated
• Original Assignee: Egnyte Incorporated
• Inventors: Wijayaratne, Ravi, Koos, Remus, White, Ray, Marathe, Manish, Tisnek, Dmitry, Krishnan, Harikesavan, Jassal, Amrit
• Primary Examiner(s): Morrison, Jay A
• Assistant Examiner(s): Caiado, Antonio J
• Application Number:
  US14/805,226
• Publication Number:
  US 20160019233A1
• Time in Patent Office: 1,484 Days
• Field of Search: 707625, 707822, 707705, 707694, 707821, 707609, 707600, 707634, 707620, 707E1701, 707E17032, 709206, 709207
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

A method for synchronizing a local file system (LFS) and a remote (cloud) file system (RFS) includes the step of storing events indicative of differences between the LFS and the RFS, where the events are associated with file system objects of the LFS and RFS. The method further includes the steps of prioritizing the events, generating file system operations for the events based at least in part on the prioritization of the events, and performing at least some of the file system operations to synchronize the LFS and RFS. Prioritizing the events includes defining a plurality of service classes (e.g., based on type of event, event attribute, etc.), assigning a priority to each of the service classes, and assigning each event to one of said service classes. Systems are also described. The invention allocates limited synchronization resources based on a predefined policy to synchronize important file system objects sooner.

Claims

1. In a local file storage system, a method for synchronizing a local file system (LFS) and a remote file system (RFS) that is located remotely from said LFS, said method comprising:

generating steady-state sync (SSS) events in response to changes being made to said LFS by a user, each of said SSS events being associated with a local file system object of said LFS that is synchronized with a corresponding remote file system object of said RFS;

generating rescan synchronization (RS) events based on a comparison between metadata associated with said LFS and metadata associated with said RFS received from a remote file storage system, each of said RS events being indicative of a difference between corresponding local and remote file system objects of said LFS and said RFS;

storing said SSS events and said RS events;

prioritizing said SSS events and said RS events for synchronization by assigning a higher priority to said SSS events than to said RS events;

generating a set of file system operations from said SSS events and said RS events based on the prioritization of said SSS events and said RS events;

applying a first subset of said set of said file system operations to said LFS to alter said LFS; and

providing a second subset of said set of said file system operations to said remote file storage system for application to said RFS; and

wherein

portions of said LFS and said RFS are synchronized as said file system operations are applied to said LFS and said RFS; and

wherein

said step of generating a set of file system operations from said SSS events and said RS events based on the prioritization of said SSS events and said RS events includes making more synchronization resources available for generating file system operations based on said SSS events than based on said RS events; and

said step of prioritizing said SSS events and said RS events includes defining a plurality of service classes, assigning a priority to each of said service classes, assigning each of said SSS events to a first service class having a first priority, and assigning each of said RS events to a second service class having a second priority lower than said first priority; and

further including at least one of

additionally prioritizing said SSS events for synchronization with respect to other SSS events within said first service class; and

additionally prioritizing said RS events for synchronization with respect to other RS events within said second service class.

2. The method of claim 1, wherein:

said step of assigning said priority to each of said service classes includes assigning a quota of synchronization resources to each of said service classes; and

said step of generating a set of file system operations from said SSS events and said RS events includes generating file system operations for each of said service classes according to said assigned quota.

3. The method of claim 2, wherein said step of assigning a quota of synchronization resources to each of said service classes includes assigning a quota of processor time.

4. The method of claim 2, wherein said step of assigning a quota of synchronization resources to each of said service classes includes assigning a number of events to process during said step of generating a set of file system operations.

5. The method of claim 1, wherein said step of defining said plurality of service classes includes defining each of said service classes based on a type of event.

6. The method of claim 1, wherein said step of generating said RS events includes:

receiving a metadata snapshot of at least a portion of said RFS from said remote file storage system;

generating a metadata snapshot of a corresponding portion of said LFS;

comparing said snapshots to detect differences between said LFS and RFS; and

generating said RS events based on said differences.

7. The method of claim 1, wherein each of said steps of additionally prioritizing said SSS events and additionally prioritizing said RS events includes additionally prioritizing said SSS events and said RS events based on at least one attribute of said file system objects associated with said SSS events and said RS events.

8. The method of claim 1, wherein said step of defining said plurality of service classes includes defining at least some of said service classes based on at least one attribute of said file system objects.

9. The method of claim 8, wherein said attribute is selected from the group consisting of file type, last modified time, ownership, and size.

10. The method of claim 1, wherein said step of generating a set of file system operations further includes:

receiving an SSS event;

searching said stored RS events to identify at least one of said stored RS events that is related to said SSS event; and

generating file system operations based on said SSS event and said at least one related RS event, whereby at least one RS event having said lower priority is processed with said received SSS event.

11. The method of claim 1, further comprising:

receiving a plurality of remote SSS events from said remote file storage system, each of said remote SSS events being indicative of at least one change to said RFS; and

assigning said remote SSS events a higher priority than said RS events.

12. The method of claim 1, further comprising storing a last valid synchronization record each time one of said file system objects is successfully synchronized.

13. The method of claim 12, wherein:

said last valid synchronization record is stored in a files table when said successfully synchronized file system object is a file; and

said last valid synchronization record is stored in a folders table when said successfully synchronized file system object is a folder.

14. A local file storage system operative to synchronize a local file system (LFS) and a remote file system (RFS) stored on a remote file storage system, said local file storage system comprising:

a data monitor operative to generate steady-state synchronization (SSS) events in response to changes being made to said LFS by a user, each of said SSS events being associated with a local file system object of said LFS that is synchronized with a corresponding remote file system object of said RFS;

a synchronizer operative to generate rescan synchronization (RS) events by comparing metadata associated with said LFS and metadata associated with said RFS received from a remote file storage system, each of said RS events being indicative of a difference between corresponding local and remote file system objects of said LFS and said RFS;

memory containing at least one database for storing said SSS events and said RS events;

an event processor operative to generate a set of file system operations from said SSS events and said RS events;

an admissions controller operative to

prioritize said SSS events and said RS events for synchronization by assigning a higher priority to said SSS events than to said RS events and

selectively provide said SSS events and said RS events to said event processor based on said priority; and

an operations handler operative to cause a first subset of said set of said file system operations to be applied to said LFS and provide a second subset of said set of said file system operations to said remote file storage system for application to said RFS, whereby portions of said LFS and said RFS are synchronized as said file system operations are applied to said LFS and said RFS; and

wherein

each of said SSS events is assigned to a first one of a plurality of service classes;

each of said RS events is assigned to a second one of said plurality of service classes;

each of said service classes is assigned a priority for synchronization;

said first service class has a higher priority than said second service class; and

said admissions controller is further operative to at least one of

additionally prioritize said SSS events for synchronization with respect to other SSS events within said first service class and

additionally prioritize said RS events for synchronization with respect to other RS events within said second service class.

15. The system of claim 14, wherein:

each of said service classes is assigned a quota of synchronization bandwidth according to said priority; and

said admissions controller is operative to provide said SSS events and said

RS events to said event processor according to said quota associated with each of said service classes.

16. The system of claim 15, wherein said quota represents a portion of total processor time for said event processor to generate said file system operations.

17. The system of claim 15, wherein said quota represents a number of events to be provided by said admissions controller to said event processor.

18. The system of claim 14, wherein each of said service classes is defined based on a type of event.

19. The system of claim 14, wherein said synchronizer is further operative to:

receive a metadata snapshot of at least a portion of said RFS from said remote file storage system;

generate a metadata snapshot of a corresponding portion of said LFS;

compare said snapshots to detect differences between said LFS and RFS; and

generate said RS events based on said differences.

20. The system of claim 14, wherein said admissions controller is operative to additionally prioritize said SSS events and additionally prioritize said RS events based on at least one attribute of said file system objects associated with said SSS events and said RS events.

21. The system of claim 14, wherein at least one of said service classes is defined based on at least one attribute of said file system objects.

22. The system of claim 21, wherein said attribute is selected from the group consisting of file type, last modified time, ownership and size.

23. The system of claim 14, wherein, for a first SSS event assigned to said first service class, said admissions controller is further operative to:

identify at least one stored RS event assigned to said second service class that is related to said first event; and

provide said first SSS event and said at least one related RS event to said event processor, whereby at least one RS event having said lower priority is processed with said SSS event.

24. The system of claim 14, wherein:

said synchronizer is further operative to receive remote SSS events associated with said RFS from said remote file storage system, each of said remote SSS events being indicative of at least one change to said RFS; and

said remote SSS events are assigned a higher priority than said RS events.

25. The system of claim 14, wherein said memory includes a last valid sync (LVS) database operative to store an LVS record each time one of said file system objects is successfully synchronized.

26. The system of claim 25, wherein:

said LVS database includes a files table and a folders table;

said LVS record is stored in said files table when said successfully synchronized file system object is a file; and

said LVS record is stored in said folders table when said successfully synchronized file system object is a folder.

27. A local file storage system storing a local file system (LFS) to be synchronized with a remote file system (RFS) stored on a remote file storage system, said local file storage system comprising:

a data monitor operative to generate steady-state synchronization (SSS) events in response to changes being made to said LFS by a user, each of said SSS events being associated with a local file system object of said LFS that is synchronized with a corresponding remote file system object of said RFS;

a synchronizer operative to generate rescan synchronization (RS) events by comparing metadata associated with said LFS and metadata associated with said RFS received from a remote file storage system, each of said RS events being indicative of a difference between corresponding local and remote file system objects of said LFS and said RFS;

memory containing at least one database for storing said SSS events and said RS events;

means for prioritizing said SSS events and said RS events for synchronization by assigning a higher priority to said SSS events than to said RS events;

means for generating a set of file system operations from said SSS events and said RS events based on the priority of said SSS events and said RS events; and

an operations handler operative to cause a first subset of said set of said file system operations to be applied to said LFS and to cause a second subset of said set of said file system operations to be provided to said remote file storage system for application to said RFS, whereby portions of said LFS and said RFS are synchronized as said file system operations are applied to said LFS and said RFS; and

wherein

each of said SSS events is assigned to a first one of a plurality of service classes;

each of said RS events is assigned to a second one of said plurality of service classes;

each of said service classes is assigned a priority for synchronization;

said first service class has a higher priority than said second service class; and

said means for prioritizing said SSS events and said RS events is further operative to at least one of

additionally prioritize said SSS events for synchronization with respect to other SSS events within said first service class and

additionally prioritize said RS events for synchronization with respect to other RS events within said second service class.

28. A non-transitory, electronically-readable storage medium having code embodied therein for causing an electronic device to perform a method for synchronizing a local file system (LFS) and a remote file system (RFS) that is located remotely from said LFS, said code causing said electronic device to:

generate steady-state sync (SSS) events in response to changes being made to said LFS by a user, each of said SSS events being associated with a local file system object of said LFS that is synchronized with a corresponding remote file system object of said RFS;

generate rescan synchronization (RS) events based on a comparison between metadata associated with said LFS and metadata associated with said RFS received from a remote file storage system, each of said RS events being indicative of a difference between corresponding local and remote file system objects of said LFS and said RFS;

store said SSS events and said RS events;

prioritize said SSS events and said RS events for synchronization by assigning a higher priority to said SSS events than to said RS events;

generate a set of file system operations from said SSS events and said RS events based on the prioritization of said SSS events and said RS events;

apply a first subset of said set of said file system operations to said LFS to alter said LFS; and

provide a second subset of said set of said file system operations to said remote file storage system for application to said RFS; and

wherein portions of said LFS and said RFS are synchronized as said file system operations are applied to said LFS and said RFS; and

wherein generating said set of file system operations from said SSS events and said

RS events based on the prioritization of said SSS events and said RS events includes using more synchronization resources for generating file system operations based on said SSS events than based on said RS events; and

prioritizing said SSS events and said RS events includes defining a plurality of service classes, assigning a priority to each of said service classes, assigning each of said SSS events to a first service class having a first priority, and assigning each of said RS events to a second service class having a second priority lower than said first priority; and

further including at least one of

additionally prioritizing said SSS events for synchronization with respect to other SSS events within said first service class; and

additionally prioritizing said RS events for synchronization with respect to other RS events within said second service class.