USING AN RPC FRAMEWORK TO FACILITATE OUT-OF-BAND DATA TRANSFERS

• US 20160112293A1
• Filed: 10/21/2014
• Published: 04/21/2016
• Est. Priority Date: 10/21/2014
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Cowling, James, Turner, James
• Granted Patent: US 9,967,310 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 9/547 Remote procedure calls [RPC]; Web services
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/133 Protocols for remote procedure calls [RPC]

Abstract

A system that stores sets of data blocks in extents located in storage devices is described. During operation, a receiving device receives, through an RPC framework, a first call asking to transfer an extent from a sending device to the receiving device. In response, the receiving device opens a port for a data connection that operates outside the RPC framework. The receiving device makes a second call, to the sending device through the RPC framework, asking to stream the extent to the port. The receiving device subsequently receives the extent from the sending device through the port and computes a checksum for the extent. The receiving device also receives a return from the second call, including a checksum for the extent computed by the sending device. If the computed checksum matches the received checksum, the receiving device returns the first call to indicate the transfer operation completed successfully.

Claims

1. A computer-implemented method, comprising:

operating a data storage system that stores sets of data blocks in extents that are located in storage devices including a sending device and a receiving device, wherein operating the data storage system includes performing the following operations on the receiving device;

receiving, through a remote procedure call (RPC) framework, a first call to transfer an extent from the sending device to the receiving device; and

in response to the first call,

opening a port for a data connection that operates outside the RPC framework,

making a second call, to the sending device through the RPC framework, wherein the second call asks to stream the extent to the port,

receiving the extent from the sending device through the port,

computing a checksum for the extent on the receiving device,

receiving a return from the second call, from the sending device through the RPC framework, wherein the return includes a checksum for the extent computed by the sending device, and

if the checksum computed on the receiving device matches the checksum computed on the sending device, returning the first call, through the RPC framework, to indicate the transfer operation completed successfully.

2. The computer-implemented method of claim 1,

wherein the data connection is a transmission control protocol (TCP) connection;

and

wherein the port is a TCP ephemeral port.

3. The computer-implemented method of claim 1, wherein the checksum comprises an MD5 hash of the extent.

4. The computer-implemented method of claim 1,

wherein after receiving the extent, the method further comprises storing the extent in a scratch extent on the receiving device, wherein the scratch extent is accessible through a private identifier;

and

wherein returning the first call involves returning the private identifier for the scratch extent, thereby enabling a requester to access the scratch extent on the receiving device.

5. The computer-implemented method of claim 4,

wherein the private identifier is randomly selected from a range of identifiers;

and

wherein the range of identifiers is sufficiently large to ensure to a specified probability that the private identifier is unguessable.

6. The computer-implemented method of claim 1, wherein the transfer of the extent is associated with moving the extent from the sending device to the receiving device.

7. The computer-implemented method of claim 1, wherein the transfer of the extent is associated with replicating the extent from the sending device to the receiving device.

8. The computer-implemented method of claim 1, wherein operating the data storage system includes performing the following operations on the sending device:

receiving, through the RPC framework, the second call asking to stream the extent from the sending device through the data connection to the port on the receiving device; and

in response to the second call,

streaming the extent through the data connection to the port on the receiving device,

computing a checksum for the extent on the sending device, and

returning the second call along with the computed checksum for the extent through the RPC framework to the receiving device, thereby enabling the receiving device to compare the checksum for the extent computed by the receiving device with the checksum for the extent computed by the sending device to determine whether the extent was transferred successfully.

9. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:

operating a data storage system that stores sets of data blocks in extents that are located in storage devices including a sending device and a receiving device, wherein operating the data storage system includes performing the following operations on the receiving device;

receiving, through a remote procedure call (RPC) framework, a first call to transfer an extent from the sending device to the receiving device; and

in response to the first call,

opening a port for a data connection that operates outside the RPC framework,

making a second call, to the sending device through the RPC framework, wherein the second call asks to stream the extent to the port,

receiving the extent from the sending device through the port,

computing a checksum for the extent on the receiving device,

receiving a return from the second call, from the sending device through the RPC framework, wherein the return includes a checksum for the extent computed by the sending device, and

if the checksum computed on the receiving device matches the checksum computed on the sending device, returning the first call, through the RPC framework, to indicate the transfer operation completed successfully.

10. The non-transitory computer-readable storage medium of claim 9,

wherein the data connection is a transmission control protocol (TCP) connection;

and

wherein the port is a TCP ephemeral port.

11. The non-transitory computer-readable storage medium of claim 9, wherein the checksum comprises an MD5 hash of the extent.

12. The non-transitory computer-readable storage medium of claim 9,

wherein after receiving the extent, the method further comprises storing the extent in a scratch extent on the receiving device, wherein the scratch extent is accessible through a private identifier;

and

wherein returning the first call involves returning the private identifier for the scratch extent, thereby enabling a requester to access the scratch extent on the receiving device.

13. The non-transitory computer-readable storage medium of claim 12,

wherein the private identifier is randomly selected from a range of identifiers;

and

wherein the range of identifiers is sufficiently large to ensure to a specified probability that the private identifier is unguessable.

14. The non-transitory computer-readable storage medium of claim 9, wherein the transfer of the extent is associated with moving the extent from the sending device to the receiving device.

15. The non-transitory computer-readable storage medium of claim 9, wherein the transfer of the extent is associated with replicating the extent from the sending device to the receiving device.

16. The non-transitory computer-readable storage medium of claim 9, wherein operating the data storage system includes performing the following operations on the sending device:

receiving, through the RPC framework, the request to stream the extent from the sending device through the data connection to the port on the receiving device; and

in response to the received request,

streaming the extent through the data connection to the port on the receiving device,

computing a checksum for the extent on the sending device, and

sending the computed checksum for the extent through the RPC framework to the receiving device, thereby enabling the receiving device to compare the checksum for the extent computed by the receiving device with the checksum for the extent computed by the sending device to determine whether the extent was transferred successfully.

17. A data storage system, comprising:

one or more storage devices including a sending device and a receiving device, wherein the one or more storage devices contain extents that store sets of data blocks; and

wherein upon receiving, through a remote procedure call (RPC) framework, a first call to transfer an extent from the sending device to the receiving device, the receiving device is configured to,

open a port for a data connection that operates outside the RPC framework,

make a second call, to the sending device through the RPC framework, wherein the second call asks to stream the extent to the port,

receive the extent from the sending device through the port,

compute a checksum for the extent on the receiving device,

receive a return from the second call, from the sending device through the RPC framework, wherein the return includes a checksum for the extent computed by the sending device, and

if the checksum computed on the receiving device matches the checksum computed on the sending device, return the first call, through the RPC framework, to indicate the transfer operation completed successfully.

18. The data storage system of claim 17,

wherein the data connection is a transmission control protocol (TCP) connection;

and

wherein the port is a TCP ephemeral port.

19. The data storage system of claim 17, wherein the checksum comprises an MD5 hash of the extent.

20. The data storage system of claim 17,

wherein after receiving the extent, the receiving device is configured to store the extent in a scratch extent on the receiving device, wherein the scratch extent is accessible through a private identifier;

and

wherein while returning the first call, the receiving device is configured to return the private identifier for the scratch extent, thereby enabling a requester to access the scratch extent on the receiving device.

21. The data storage system of claim 20,

wherein the private identifier is randomly selected from a range of identifiers;

and

wherein the range of identifiers is sufficiently large to ensure to a specified probability that the private identifier is unguessable.

22. The data storage system of claim 17, wherein the transfer of the extent is associated with moving the extent from the sending device to the receiving device.

23. The data storage system of claim 17, wherein the transfer of the extent is associated with replicating the extent from the sending device to the receiving device.

24. The data storage system of claim 17, wherein upon receiving, through the RPC framework, the second call asking to stream the extent from the sending device through the data connection to the port on the receiving device, the sending device is configured to:

stream the extent through the data connection to the port on the receiving device,

compute a checksum for the extent on the sending device, and

send the second call along with the computed checksum for the extent through the RPC framework to the receiving device, thereby enabling the receiving device to compare the checksum for the extent computed by the receiving device with the checksum for the extent computed by the sending device to determine whether the extent was transferred successfully.