Content item block replication protocol for multi-premises hosting of digital content items

• US 9,697,269 B2
• Filed: 12/22/2015
• Issued: 07/04/2017
• Est. Priority Date: 10/29/2015
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Koorapati, Nipunn, Rude, Christopher, von Muhlen, Marcio, Bunger, Nils
• Primary Examiner(s): ALAM, SHAHID AL
• Application Number:
  US14/979,268
• Publication Number:
  US 20170126800A1
• Time in Patent Office: 560 Days
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: H04L 47/196 Integration of transport layer protocols, e.g. TCP and UDP
• CPC Class Codes: H04L 61/457 containing identifiers of data entities on a computer, e.g. file names
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/108 characterised by resources being split in blocks or fragments
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

A content item block replication protocol for multi-premises hosting of digital content items. In one embodiment, for example, a method comprises: receiving, from a server, a server journal entry identifying one or more content item blocks of a content item represented by the server journal entry; storing a replication task log entry corresponding to the server journal entry in a replication task log, the replication task log entry identifying the one or more content item blocks of the content item represented by the server journal entry and identifying a block server; and either offering to send the one or more content item blocks identified in the replication task log entry to the block server identified in the replication task log entry, or downloading the one or more content item blocks identified in the replication task log entry from the block server identified in the replication task log entry.

Claims

1. A method, comprising:

at a computing device comprising one or more processors and storage media storing one or more computer programs executed by the one or more processors to perform the method, performing operations comprising;

receiving, from a server, a server journal entry, the server journal entry identifying a plurality of content item blocks of a content item represented by the server journal entry;

storing a replication task log entry corresponding to the server journal entry in a replication task log, the replication task log entry identifying the plurality of content item blocks of the content item represented by the server journal entry and identifying an on-premises block server;

offering to send the plurality of content item blocks identified in the replication task log entry to the on-premises block server identified in the replication task log entry;

receiving, from the on-premises block server, an acceptance of the offer to send the plurality of content item blocks, the acceptance identifying less than all of the plurality of content item blocks;

sending less than all of the plurality of content item blocks to the on-premises block server; and

updating metadata of the replication task log entry to indicate replication task completion after successfully sending less than all of the plurality of content item blocks to the on-premises block server.

2. The method of claim 1, the operations further comprising:

prior to receiving the server journal entry from the server, sending, to the server, a current client cursor value for a content item namespace to which the content item belongs;

receiving the server journal entry in response sending the current client cursor value, the server journal entry comprising a server journal cursor value for the content item namespace; and

updating the current client cursor value for the content item namespace to equal the server journal cursor value for the content item namespace from the server journal entry.

3. The method of claim 1, wherein the on-premises block server is connected to the same local area network as the computing device.

4. The method of claim 1, wherein:

the server journal entry is a first server journal entry;

the replication task log entry is a first replication task log entry;

the content item is a first content item;

the operations further comprise;

receiving, from a server, a second server journal entry that is not the first server journal entry, the second server journal entry identifying one or more content item blocks of a second content item, that is not the first content item, represented by the second server journal entry;

storing a second replication task log entry, that is not the first replication task log entry, corresponding to the second server journal entry in the replication task log, the second replication task log entry identifying the one or more content item blocks of the second content item represented by the second server journal entry and identifying an off-premises block server of an online content management service; and

downloading the one or more content item blocks identified in the second replication task log entry from the off-premises block server identified in the second replication task log entry.

5. The method of claim 4, the operations further comprising:

updating metadata of the second replication task log entry to indicate replication task completion after successfully downloading the one or more content item blocks from the off-premises block server.

6. The method of claim 4, wherein:

the second server journal entry identifies a plurality of content item blocks of the second content item represented by the second server journal entry;

the second replication task log entry identifies the plurality of content item blocks of the second content item represented by the second server journal entry;

the operations further comprise determining that at least one of the plurality of content item blocks of the second content item is already stored at the computing device; and

wherein the one or more content item blocks downloaded from the off-premises block server does not include the at least one content item block already stored at the computing device.

7. The method of claim 4, the operations further comprising:

prior to receiving the second server journal entry from the server, sending, to the server, a current client cursor value for a content item namespace to which the second content item belongs;

receiving the second server journal entry in response sending the current client cursor value for the content item namespace, the second server journal entry comprising a server journal cursor value for the content item namespace; and

updating the current client cursor value for the content item namespace to equal the server journal cursor value for the content item namespace from the server journal entry.

8. The method of claim 4, the operations further comprising:

serving the one or more content item blocks of the second content item to one or more personal computing devices that request to download the one or more content item blocks.

9. A computing device, comprising:

one or more processors; and

storage media storing one or more computer programs comprising instructions configured for;

receiving, from a server, a server journal entry, the server journal entry identifying a plurality of content item blocks of a content item represented by the server journal entry;

storing a replication task log entry corresponding to the server journal entry in a replication task log, the replication task log entry identifying the plurality of content item blocks of the content item represented by the server journal entry and identifying an on-premises block server;

offering to send the plurality of content item blocks identified in the replication task log entry to the on-premises block server identified in the replication task log entry;

receiving, from the on-premises block server, an acceptance of the offer to send the plurality of content item blocks, the acceptance identifying less than all of the plurality of content item blocks;

sending less than all of the plurality of content item blocks to the on-premises block server; and

updating metadata of the replication task log entry to indicate replication task completion after successfully sending less than all of the plurality of content item blocks to the on-premises block server.

10. The computing device of claim 9, the instructions further configured for:

prior to receiving the server journal entry from the server, sending, to the server, a current client cursor value for a content item namespace to which the content item belongs;

receiving the server journal entry in response sending the current client cursor value, the server journal entry comprising a server journal cursor value for the content item namespace; and

updating the current client cursor value for the content item namespace to equal the server journal cursor value for the content item namespace from the server journal entry.

11. The computing device of claim 9, wherein the on-premises block server is connected to the same local area network as the computing device.

12. The computing device of claim 9, wherein:

the server journal entry is a first server journal entry;

the replication task log entry is a first replication task log entry;

the content item is a first content item;

the instructions further configured for;

receiving, from a server, a second server journal entry that is not the first server journal entry, the second server journal entry identifying one or more content item blocks of a second content item, that is not the first content item, represented by the second server journal entry;

storing a second replication task log entry, that is not the first replication task log entry, corresponding to the second server journal entry in the replication task log, the second replication task log entry identifying the one or more content item blocks of the second content item represented by the second server journal entry and identifying an off-premises block server of an online content management service; and

downloading the one or more content item blocks identified in the second replication task log entry from the off-premises block server identified in the second replication task log entry.

13. The computing device of claim 12, the instructions further configured for:

updating metadata of the second replication task log entry to indicate replication task completion after successfully downloading the one or more content item blocks from the off-premises block server.

14. The computing device of claim 12, wherein:

the second server journal entry identifies a plurality of content item blocks of the content item represented by the second server journal entry;

the second replication task log entry identifies the plurality of content item blocks of the content item represented by the second server journal entry;

the instructions further configured for determining that at least one of the plurality of content item blocks of the second content item is already stored at the computing device; and

wherein the one or more content item blocks downloaded from the off-premises block server does not include the at least one content item block already stored at the computing device.

15. The computing device of claim 12, the instructions further configured for:

prior to receiving the second server journal entry from the server, sending, to the server, a current client cursor value for a content item namespace to which the second content item belongs;

receiving the second server journal entry in response sending the current client cursor value for the content item namespace, the second server journal entry comprising a server journal cursor value for the content item namespace; and

updating the current client cursor value for the content item namespace to equal the server journal cursor value for the content item namespace from the server journal entry.

16. The computing device of claim 12, the instructions further configured for:

serving the one or more content item blocks of the second content item to one or more personal computing devices that request to download the one or more content item blocks.