Content item block replication protocol for multi-premises hosting of digital content items

• US 10,133,804 B2
• Filed: 05/23/2017
• Issued: 11/20/2018
• Est. Priority Date: 10/29/2015
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Koorapati, Nipunn, Rude, Christopher, von Muhlen, Marcio, Bunger, Nils
• Primary Examiner(s): Alam, Shahid
• Application Number:
  US15/603,193
• Publication Number:
  US 20170255687A1
• Time in Patent Office: 546 Days
• Field of Search: 707620
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: H04L 47/196 Integration of transport layer protocols, e.g. TCP and UDP
• CPC Class Codes: H04L 61/457 containing identifiers of data entities on a computer, e.g. file names
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/108 characterised by resources being split in blocks or fragments
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

A content item block replication protocol for multi-premises hosting of digital content items. In one embodiment, for example, a method comprises: receiving, from a server, a server journal entry identifying one or more content item blocks of a content item represented by the server journal entry; storing a replication log entry corresponding to the server journal entry in a replication log, the replication log entry identifying the one or more content item blocks of the content item represented by the server journal entry and identifying a block server; and either offering to send the one or more content item blocks identified in the replication log entry to the block server identified in the replication log entry, or downloading the one or more content item blocks identified in the replication log entry from the block server identified in the replication log entry.

Claims

1. A method for a content item block replication protocol, the method performed by an on-premises block server comprising one or more processors and memory, the method comprising:

receiving, from a metadata server of an online content management service by a data network communicatively coupling the on-premises block server to the metadata server, an identification of content item blocks, wherein a content item, hosted with the online content management service, comprises the content item blocks;

based, at least in part, on the received identification, storing a replication log entry in a replication log, the replication log entry comprising the identification of content item blocks and an identifier of a peer on-premises content item block server;

based, at least in part, on processing the replication log entry, transmitting, to the peer on-premises content item block server identified in the replication log entry by a data network communicatively coupling the on-premises block server and the peer on-premises block server, an offer to send at least one content item block, of the content item blocks of which the content item is composed, to the peer on-premises content item block server identified in the replication log entry;

receiving, from the peer on-premises content item block server by the data network communicatively coupling the on-premises block server and the peer on-premises block server, an acceptance of the offer to send the at least one content item block; and

based, at least in part, on the received acceptance, sending, by the data network communicatively coupling the on-premises block server and the peer on-premises block server, the at least one content item block to the peer on-premises content item block server.

2. The method of claim 1, further comprising:

based, at least in part, on completing the processing of the replication log entry, updating the replication log entry to indicate the replication log entry is complete.

3. The method of claim 1, wherein all of the content item blocks of which the content item is composed are offered to the peer on-premises content item block server identified in the replication log entry.

4. The method of claim 1, wherein the acceptance identifies one or more content item blocks, of the content item blocks of which the content item is composed, to send;

wherein the identified one or more content item blocks do not include all of the content item blocks of which the content item is composed; and

wherein the method further comprises sending the identified one or more content item blocks to the peer on-premises content item block server.

5. The method of claim 1, wherein:

the peer on-premises content item block server is a first peer on-premises content item block server; and

the method further comprises, prior to the processing the replication log entry, downloading at least one of the content item blocks, of which the content item is composed, from a second content item block server that is not the first peer on-premises content item block server.

6. The method of claim 1, further comprising:

determining a current client journal cursor value associated with a content item namespace;

sending the current client journal cursor value and an identifier of the associated content item namespace to the metadata sever; and

wherein the received identification of content item blocks, of which the content item is composed, is based, at least in part, on the sent current client journal cursor value and the content item namespace identifier.

7. The method of claim 1, further comprising:

determining that a content item namespace is assigned to the peer on-premises content item block server, the content item belonging to the content item namespace; and

wherein the offering to send the at least one content item block, of the content item blocks of which the content item is composed, to the peer on-premises content item block server is based, at least in part, on the determining that the content item namespace is assigned to the peer on-premises content item block server.

8. The method of claim 1, wherein:

the peer on-premises content item block server is a first peer on-premises content item block server;

the replication log entry comprises an identifier of a second peer on-premises content item block server that is not the first peer on-premises content item block server; and

the method further comprises offering to send the at least one content item block, of the content item blocks of which the content item is composed, to the second peer on-premises content item block server identified in the replication log entry, based, at least in part, on the processing the replication log entry.

9. The method of claim 1, wherein:

the peer on-premises content item block server is a first peer on-premises content item block server;

the content item belongs to a content item namespace;

the method further comprises;

determining a second peer on-premises content item block server that is not the first peer on-premises content item block server to which the content item namespace is assigned;

determining that a peering relationship does not exist with between the on-premises block server and the second peer on-premises content item block server; and

determining not to offer to send to the second peer on-premises content item block server any of the content item blocks, of which the content item is composed, based, at least in part, on the determining that the peering relationship does not exist; and

the replication log entry does not identify the second peer on-premises content item block server.

10. The method of claim 1, wherein:

the method further comprises identifying, for deletion according to a least recently used (LRU) criterion, one or more content item blocks that are locally stored by the on-premises block server; and

the LRU criterion is based on content item block download time, content item block upload time, or content item block download time and content item block upload time.

11. The method of claim 1, wherein the receiving, from the metadata server, the identification of content item blocks, of which the content item is composed, is based, at least in part, on an assignment of a content item namespace to the peer on-premises content item block server.

12. An on-premises block server comprising:

one or more processors;

one or more programs for a content item block replication protocol, the one or more programs configured for execution by the one or more processors, the one or more programs comprising instructions configured for;

receiving, from a metadata server of an online content management service by a data network communicatively coupling the on-premises block server to the metadata server, an identification of content item blocks, wherein a content item hosted with the online content management service comprises the content item blocks;

based, at least in part, on the received identification, storing a replication log entry in a replication log, the replication log entry comprising the identification of content item blocks and an identifier of a peer on-premises content item block server; and

based, at least in part, on processing the replication log entry, transmitting, to the peer on-premises content item block server identified in the replication log entry by a data network communicatively coupling the on-premises block server and the peer on-premises block server, an offer to send at least one content item block, of the content item blocks of which the content item is composed, to the peer on-premises content item block server identified in the replication log entry;

receiving, from the peer on-premises content item block server, an acceptance of the offer to send the at least one content item block; and

based, at least in part, on the received acceptance, sending, by the data network communicatively coupling the on-premises block server and the peer on-premises block server, the at least one content item block to the peer on-premises content item block server.

13. The on-premises block server of claim 12, the instructions further configured for:

based, at least in part, on completing the processing of the replication log entry, updating the replication log entry to indicate the replication log entry is complete.

14. The on-premises block server of claim 12, wherein all of the content item blocks of which the content item is composed are offered to the peer on-premises content item block server identified in the replication log entry.

15. The on-premises block server of claim 12, wherein the acceptance identifies one or more content item blocks, of the content item blocks of which the content item is composed, to send;

wherein the identified one or more content item blocks do not include all of the content item blocks of which the content item is composed; and

wherein the instructions are further configured for;

sending the identified one or more content item blocks to the peer on-premises content item block server.

16. The on-premises block server of claim 12, wherein:

the peer on-premises content item block server is a first peer on-premises content item block server; and

the instructions are further configured for;

prior to the processing the replication log entry, downloading at least one of the content item blocks, of which the content item is composed, from a second content item block server that is not the first peer on-premises content item block server.

17. The on-premises block server of claim 12, the instructions further configured for:

determining a current client journal cursor value associated with a content item namespace;

sending the current client journal cursor value and an identifier of the associated content item namespace to the metadata sever; and

wherein the received identification of content item blocks, of which the content item is composed, is based, at least in part, on the sent current client journal cursor value and the content item namespace identifier.