Distributed file system backup and synchronization

• US 9,904,604 B2
• Filed: 12/31/2014
• Issued: 02/27/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Crashplan Group LLC
• Original Assignee: Code 42 Software Incorporated
• Inventors: Dornquast, Matthew, Bispala, Brian, Scorcio, Marshall, Armstrong, Brad, Lindquist, Peter, Parker, Christopher
• Primary Examiner(s): Le, Debbie
• Application Number:
  US14/587,063
• Publication Number:
  US 20160092444A1
• Time in Patent Office: 1,154 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/1453 using de-duplication of the data
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/1873 Versioning file systems, temporal file systems, e.g. file system supporting different historic versions of files
• CPC Class Codes: G06F 16/2329 using versioning
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/604 Tools and structures for managing or administering access control systems
• CPC Class Codes: G06F 2221/2111 Location-sensitive, e.g. geographical location, GPS
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 47/821 Prioritising resource allocation or reservation requests
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 67/1072 Discovery involving ranked list compilation of candidate peers
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

Systems and techniques for a distributed file system element collection are described herein. A node may identify a state of a file system element, which is identified in a file system element definition, from a local data store. The state corresponds to a distribution action. The file system element definition is one of a plurality of file system element definitions in a collection schema of a collection to which the node is a member. The node may, in response to identifying the state of the file system element, issue a communication to some members of the collection in furtherance of completion of the distribution action. A response may be received by the node from a participant node in the list of participant nodes. The node may complete the distribution action using content from the response.

Claims

1. At least one machine readable medium that is not a transitory propagating signal, the machine readable medium including instructions that, when executed by hardware of a node, cause the node to perform operations comprising:

identifying, from a local data store of the node, a state of a file system element identified in a file system element definition, the state being one of a plurality of states and corresponding to a data distribution action, the file system element definition being one of a plurality of file system element definitions in a collection schema of a collection, wherein a file system element is a file or directory of the local data store, wherein a collection is a portion of the local data store identified for distribution, wherein the portion includes at least two file system elements having different local root paths in the local data store, and wherein a collection schema defines the file system elements in the collection;

receiving a portion of the collection schema from an authority node of the collection that is distinct from the node, the portion including the plurality of file system element definitions and the authority node managing the collection schema;

issuing, by the node in response to identifying the state, a communication to a list of participant nodes in furtherance of completion of the data distribution action, the list of participant nodes being a subset of a plurality of nodes that are members of the collection;

receiving a response to the communication from a participant node in the list of participant nodes; and

completing the data distribution action using content from the response.

2. The machine readable medium of claim 1, wherein:

the state of the file system element indicates that the file system element is not current in the local data store;

the data distribution action includes making the file system element current in the local data store;

the communication is a request for contents of a current version of the file system element;

the response to the communication includes at least a portion of the contents; and

completing the data distribution action includes using the at least a portion of the contents to update the file system element on the local data store to the current version of the file system element.

3. The machine readable medium of claim 2, wherein the at least a portion of the contents are incremental differences between the current version of the file system element and the file system element in the local data store.

4. The machine readable medium of claim 3, wherein the smallest a metric upon which the incremental differences are assessed is a block.

5. The machine readable medium of claim 4, wherein using the at least a portion of the contents to update the file system element on the local data store includes replacing blocks in the file system element on the local data store with corresponding blocks in the at least a portion of the contents.

6. The machine readable medium of claim 1, wherein:

the state of the file system element indicates that the file system element has changed in the local data store;

the data distribution action includes notifying the list of participant nodes about the file system element change;

the communication is an event published to the list of participant nodes;

the response is a request for contents of the file system element; and

completing the data distribution action includes transmitting the contents to the participant node.

7. The machine readable medium of claim 6, wherein the request for contents includes identification of current contents for the file system element on the participant node, and wherein transmitting the contents includes transmitting the difference between the current contents and the contents of the file system element on the local data store.

8. The machine readable medium of claim 1, wherein a first file system element definition in the plurality of file system element definitions corresponds to a file and a second file system element definition in the plurality of file system element definitions corresponds to a directory.

9. The machine readable medium of claim 8, wherein contents of a directory file system element include files and directories within the directory file system element.

10. The machine readable medium of claim 9, wherein files within the directory file system element and part of a second collection are excluded from the contents of the directory file system element.

11. The machine readable medium of claim 1, wherein the operations further comprise synchronizing an event stream with the list of participant nodes, the event stream including indications of changes of file system elements between nodes.

12. The machine readable medium of claim 11, wherein identifying the state of a file system element on the local data store includes receiving an event from the event stream indicating an update to the file system element and comparing the indicated update to the file system element in the local data store.

13. The machine readable medium of claim 12, wherein comparing the indicated update includes comparing a file system element version between the indicated update and the local data store.

14. The machine readable medium of claim 11, wherein issuing the communication to the list of participant nodes to complete the data distribution action includes publishing an event to the event stream indicating the changed state of the file system element on the local data store.

15. The machine readable medium of claim 1, wherein the operations further comprise:

observing a user action point, the user action point to receive user input;

providing a user interface to a user to create a collection in response to receipt of the user input at the user action point; and

receiving an indication from the user to create the collection via the user interface.

16. The machine readable medium of claim 15, wherein the user action point is at least one of:

a user interface element added to a file browser application, a user interface element provided by a data distribution mechanism, or a web interface.

17. The machine readable medium of claim 15, wherein the operations further comprise:

identifying a collection type based on the indication and a context of the user interface; and

communicating the collection type, the indication, and the context of the user interface to an authority node distinct from the node, the authority node managing the collection schema.

18. The machine readable medium of claim 1, wherein the collection is a collection type that is any one of a backup collection, a personal collection, or a multi-user collection.

19. The machine readable medium of claim 17, wherein the collection type is a multi-user collection;

wherein a second collection has;

a second collection type that is a backup collection, the node being a member of the backup collection;

a list of participating backup nodes that are a subset of a plurality of backup members of the backup collection, the participant node being in the list of participating backup nodes;

a backup file system element definition; and

wherein the operations further comprise;

identifying, from the local data store, a backup state of a backup file system element identified in the backup file system element definition, the backup state being one of the plurality of states and corresponding to a backup distribution action;

issuing a second communication to the list of participating backup nodes to complete the backup distribution action;

receiving a second response to the second communication from the participant node; and

completing the backup distribution action using content from the second response.

20. The machine readable medium of claim 19, wherein the operations further comprise synchronizing an event stream with the participant node, the event stream including indications of changes of the file system elements and backup file system elements between nodes.

21. The machine readable medium of claim 20, wherein identifying the state of a file system element or the backup state of the backup file system element on the local data store includes receiving an event from the event stream indicating an update to the file system element or the backup file system element respectively and comparing the indicated update to the file system element or the backup file system element in the local data store.

22. The machine readable medium of claim 21, wherein comparing the indicated update includes comparing a file system element version between the indicated update and the local data store.

23. The machine readable medium of claim 20, wherein issuing the communication to the list of participant nodes or the second communication to the list of participating backup nodes to complete the distribution action or the backup distribution action includes publishing an event to the event stream indicating the changed state of the file system element or the changed backup state of the backup file system element on the local data store.

24. The machine readable medium of claim 1, wherein each of the plurality of file system element definitions include a local root corresponding to each member of the collection.

25. The machine readable medium of claim 24, wherein a local root corresponding to the node is different than a local root corresponding to the participant node for the file system element definition corresponding to the file system element.