Using an RPC framework to facilitate out-of-band data transfers

• US 9,967,310 B2
• Filed: 10/21/2014
• Issued: 05/08/2018
• Est. Priority Date: 10/21/2014
• Status: Active Grant

• Current Assignee: Dropbox, Inc.
• Original Assignee: Dropbox, Inc.
• Inventors: Cowling, James, Turner, James
• Primary Examiner(s): Nguyen, Minh Chau
• Application Number:
  US14/519,888
• Publication Number:
  US 20160112293A1
• Time in Patent Office: 1,295 Days
• Field of Search: 709223-226
• CPC Class Codes: G06F 9/547 Remote procedure calls [RPC]; Web services
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/133 Protocols for remote procedure calls [RPC]

Abstract

A system that stores sets of data blocks in extents located in storage devices is described. During operation, a receiving device receives, through an RPC framework, a first call asking to transfer an extent from a sending device to the receiving device. In response, the receiving device opens a port for a data connection that operates outside the RPC framework. The receiving device makes a second call, to the sending device through the RPC framework, asking to stream the extent to the port. The receiving device subsequently receives the extent from the sending device through the port and computes a checksum for the extent. The receiving device also receives a return from the second call, including a checksum for the extent computed by the sending device. If the computed checksum matches the received checksum, the receiving device returns the first call to indicate the transfer operation completed successfully.

Claims

1. A computer-implemented method, comprising:

a receiving device receiving, through a remote procedure call (RPC) framework, a first call to transfer an extent from a sending device to the receiving device, the extent containing data blocks;

based on the first call;

the receiving device opening an ephemeral port for an unauthenticated data connection that operates outside the RPC framework, and

the receiving device making a second call, to the sending device through the RPC framework, wherein the second call asks to stream the extent to the ephemeral port, the second call identifying the ephemeral port to which to stream the extent;

the receiving device receiving the extent from the sending device through the unauthenticated data connection and the ephemeral port;

computing a checksum for the extent on the receiving device;

the receiving device receiving a return from the second call, from the sending device through the RPC framework, wherein the return includes a checksum for the extent computed by the sending device; and

based, at least in part, on determining that the checksum computed on the receiving device matches the checksum computed on the sending device, returning the first call through the RPC framework.

2. The computer-implemented method of claim 1, comprising:

after receiving the extent, storing the extent in a scratch extent on the receiving device, wherein the scratch extent is accessible through a private identifier; and

wherein returning the first call involves returning the private identifier for the scratch extent, thereby enabling a requester to access the scratch extent on the receiving device.

3. The computer-implemented method of claim 2,

wherein the private identifier is randomly selected from a range of identifiers;

and

wherein the range of identifiers is sufficiently large to ensure to a specified probability that the private identifier is unguessable.

4. The computer-implemented method of claim 1,

wherein the unauthenticated data connection is a transmission control protocol (TCP) connection;

and

wherein the ephemeral port is a TCP ephemeral port.

5. The computer-implemented method of claim 1, wherein the checksum computed on the receiving device comprises an MD5 hash of the extent.

6. The computer-implemented method of claim 1, wherein the transfer of the extent is associated with moving the extent from the sending device to the receiving device.

7. The computer-implemented method of claim 1, wherein the transfer of the extent is associated with replicating the extent from the sending device to the receiving device.

8. The computer-implemented method of claim 1, comprising performing, on the sending device:

receiving, through the RPC framework, the second call asking to stream the extent from the sending device through the unauthenticated data connection to the ephemeral port on the receiving device; and

in response to the second call;

streaming the extent through the unauthenticated data connection to the ephemeral port on the receiving device,

computing the checksum for the extent on the sending device, and

returning the second call along with the checksum, computed on the sending device, for the extent through the RPC framework to the receiving device, thereby enabling the receiving device to compare the checksum for the extent computed by the receiving device with the checksum for the extent computed by the sending device to determine whether the extent was transferred successfully.

9. One or more non-transitory computer-readable media storage medium storing instructions that when executed by a computer system, cause the computer system to perform:

a receiving device receiving, through a remote procedure call (RPC) framework, a first call to transfer an extent from a sending device to the receiving device, the extent containing data blocks;

based on the first call;

the receiving device opening an ephemeral port for an unauthenticated data connection that operates outside the RPC framework, and

the receiving device making a second call, to the sending device through the RPC framework, wherein the second call asks to stream the extent to the ephemeral port, the second call identifying the ephemeral port to which to stream the extent;

the receiving device receiving the extent from the sending device through the unauthenticated data connection and the ephemeral port;

computing a checksum for the extent on the receiving device;

the receiving device receiving a return from the second call, from the sending device through the RPC framework, wherein the return includes a checksum for the extent computed by the sending device; and

based, at least in part, on determining that the checksum computed on the receiving device matches the checksum computed on the sending device, returning the first call through the RPC framework.

10. The one or more non-transitory computer-readable media of claim 9, wherein the instructions further comprise instructions that, when executed by the computer system, cause the computer system to perform:

after receiving the extent, storing the extent in a scratch extent on the receiving device, wherein the scratch extent is accessible through a private identifier; and

wherein returning the first call involves returning the private identifier for the scratch extent, thereby enabling a requester to access the scratch extent on the receiving device.

11. The one or more non-transitory computer-readable media of claim 10,

wherein the private identifier is randomly selected from a range of identifiers;

and

wherein the range of identifiers is sufficiently large to ensure to a specified probability that the private identifier is unguessable.

12. The one or more non-transitory computer-readable media of claim 9,

wherein the unauthenticated data connection is a transmission control protocol (TCP) connection;

and

wherein the ephemeral port is a TCP ephemeral port.

13. The one or more non-transitory computer-readable media of claim 9, wherein the checksum computed on the receiving device comprises an MD5 hash of the extent.

14. The one or more non-transitory computer-readable media of claim 9, wherein the transfer of the extent is associated with moving the extent from the sending device to the receiving device.

15. The one or more non-transitory computer-readable media of claim 9, wherein the transfer of the extent is associated with replicating the extent from the sending device to the receiving device.

16. The one or more non-transitory computer-readable media of claim 9, wherein the instructions further comprise instructions that, when executed by the computer system, cause the computer system to perform on the sending device:

receiving, through the RPC framework, the second call asking to stream the extent from the sending device through the unauthenticated data connection to the ephemeral port on the receiving device; and

in response to the second call;

streaming the extent through the unauthenticated data connection to the ephemeral port on the receiving device,

computing the checksum for the extent on the sending device, and

returning the second call along with the checksum, computed on the sending device, for the extent through the RPC framework to the receiving device, thereby enabling the receiving device to compare the checksum for the extent computed by the receiving device with the checksum for the extent computed by the sending device to determine whether the extent was transferred successfully.

17. A system, comprising:

one or more processors;

one or more non-transitory media storing instructions configured for execution by the one or more processors, the instructions configured to perform;

a receiving device receiving, through a remote procedure call (RPC) framework, a first call to transfer an extent from a sending device to the receiving device, the extent containing data blocks;

based on the first call;

the receiving device opening an ephemeral port for an unauthenticated data connection that operates outside the RPC framework and

the receiving device making a second call, to the sending device through the RPC framework, wherein the second call asks to stream the extent to the ephemeral port, the second call identifying the ephemeral port to which to stream the extent;

the receiving device receiving the extent from the sending device through the unauthenticated data connection and the ephemeral port;

computing a checksum for the extent on the receiving device;

the receiving device receiving a return from the second call, from the sending device through the RPC framework, wherein the return includes a checksum for the extent computed by the sending device; and

based, at least in part, on determining that the checksum computed on the receiving device matches the checksum computed on the sending device, returning the first call through the RPC framework.

18. The system of claim 17, wherein the instructions further comprise instructions that are configured to perform:

after receiving the extent, storing the extent in a scratch extent on the receiving device, wherein the scratch extent is accessible through a private identifier; and

wherein returning the first call involves returning the private identifier for the scratch extent, thereby enabling a requester to access the scratch extent on the receiving device.

19. The system of claim 18,

wherein the private identifier is randomly selected from a range of identifiers;

and

wherein the range of identifiers is sufficiently large to ensure to a specified probability that the private identifier is unguessable.

20. The system of claim 17,

wherein the unauthenticated data connection is a transmission control protocol (TCP) connection;

and

wherein the ephemeral port is a TCP ephemeral port.

21. The system of claim 17, wherein the checksum computed on the receiving device comprises an MD5 hash of the extent.

22. The system of claim 17, wherein the transfer of the extent is associated with moving the extent from the sending device to the receiving device.

23. The system of claim 17, wherein the transfer of the extent is associated with replicating the extent from the sending device to the receiving device.

24. The system of claim 17, wherein the instructions further comprise instructions that are configured to perform on the sending device:

receiving, through the RPC framework, the second call asking to stream the extent from the sending device through the unauthenticated data connection to the ephemeral port on the receiving device; and

in response to the second call;

streaming the extent through the unauthenticated data connection to the ephemeral port on the receiving device,

computing the checksum for the extent on the sending device, and

returning the second call along with the checksum, computed on the sending device, for the extent through the RPC framework to the receiving device, thereby enabling the receiving device to compare the checksum for the extent computed by the receiving device with the checksum for the extent computed by the sending device to determine whether the extent was transferred successfully.