DATA TRANSFER OPTIMIZATIONS

• US 20150156174A1
• Filed: 12/03/2013
• Published: 06/04/2015
• Est. Priority Date: 12/03/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Fahey, Sean Anthony, Hill, Brent James
• Granted Patent: US 10,187,358 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/18 using different networks or channels, e.g. using out of band channels cryptographic mechanisms or cryptographic arrangements for key distribution involving distinctive intermediate devices or communication paths H04L9/0827; cryptographic mechanisms or cryptographic arrangements for authentication using a plurality of channels H04L9/3215
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Data transfer between a first computer system and a second computer system utilize parallel servers of the second computer system. A plurality of data chunks collectively comprise a data object. The data chunks may be encrypted and sent over parallel channels to the second computer system, which may be a data storage service of a computing resource service provider. The data object, or a portion thereof, may be downloaded from the data storage system in parallel.

Claims

1. A computer-readable storage medium having stored thereon instructions that, when executed by one or more processors of a computer system, cause the computer system to:

deconstruct a data object into a plurality of data chunks a plurality of which are of uniform size;

separately encrypt each of the plurality of data chunks with the same cryptographic key such that at least a first subset of the plurality of data chunks are encrypted in parallel using a different initialization vector for each chunk in the first subset of the plurality of data chunks; and

transmit each of the plurality of separately encrypted data chunks to a data storage service such that at least a second subset of the plurality of separately encrypted data chunks are transmitted in parallel to multiple servers of the data storage service and such that the plurality of separately encrypted data chunks are collectively retrievable from the data storage service using an identifier for the plurality of separately encrypted data chunks.

2. The computer-readable storage medium of claim 1, wherein transmitting each of the plurality of separately encrypted data chunks includes, for each separately encrypted data chunk of the plurality of separately encrypted data chunks, transmitting with the separately encrypted data chunk a corresponding initialization vector used to obtain the separately encrypted data chunk in encrypted form.

3. The computer-readable storage medium of claim 1, wherein the instructions further cause the computer system to:

separately download, from the data storage service, each of the separately encrypted data chunks such that at least a third subset of the separately encrypted data chunks are downloaded in parallel; and

decrypt each of the separately downloaded separately encrypted data chunks such that at least a fourth subset of the separately downloaded separately encrypted data chunks are encrypted in parallel.

4. The computer-readable storage medium of claim 3, wherein separately downloading each of the separately encrypted data chunks includes, for each separately encrypted data chunk of the plurality of separately encrypted data chunks, transmitting a request to the data storage service that specifies a range of bytes corresponding to the separately encrypted data chunk.

5. The computer-readable storage medium of claim 1, wherein:

the data object is a media file;

the instructions further cause the computer system to;

receive user input indicating a playback start time different from a beginning time of the media file;

separately download, from the data storage service, a proper subset of the separately encrypted data chunks corresponding to a portion of the media file that includes a range of the media file that includes the start time and lacks the beginning time.

6. A computer-readable storage medium having stored thereon instructions that, when executed by one or more processors of a computer system, cause the computer system to:

obtain a plurality of data chunks;

cause each of the plurality of data chunks to be encrypted such that at least a first subset of the plurality of data chunks are encrypted in parallel using a different initialization vector for each chunk in the first subset of the plurality of data chunks; and

transmit each of the plurality of separately encrypted data chunks to a data storage service such that at least a second subset of the plurality of separately encrypted data chunks are transmitted in parallel to multiple servers of the data storage service and that the plurality of separately encrypted data chunks are collectively managed by the data storage service as a data object.

7. The computer-readable storage medium of claim 6, wherein, causing each of the plurality of data chunks to be encrypted includes causing each of the plurality of data chunks to be encrypted under the same cryptographic key.

8. The computer-readable storage medium of claim 6, wherein transmitting each of the plurality of separately encrypted data chunks to the data storage service includes, for each separately encrypted data chunk of the plurality of separately encrypted data chunks, transmitting a sub-object of the data object that includes both the separately encrypted data chunk and a corresponding initialization vector used to obtain the separately encrypted data chunk.

9. The computer-readable storage medium of claim 6, wherein, the instructions further cause the computer system to download, from the data storage service, at least a subset of the plurality of separately encrypted data chunks by, for each separately encrypted data chunk of the plurality of separately encrypted data chunks, submitting a request to the data storage service that specifies a range of data within the data object that contains the separately encrypted data chunk.

10. The computer-readable storage medium of claim 9, wherein downloading the subset of the plurality of separately encrypted data chunks includes determining, based on a uniform data chunk size of at least some of the data chunks in the subset and a data object size of the data object, a plurality of ranges each corresponding to a separately encrypted data chunk from the subset.

11. A system, comprising:

a plurality of processors; and

memory encoding instructions executable by at least one of the processors to cause the system to;

manipulate each data chunk of a plurality of data chunks that collectively comprise a data object by distributing, among the plurality of processors, the plurality of data chunks for manipulation such that each subset of a plurality of subsets are manipulated using a different initialization vector; and

transfer the plurality of manipulated data chunks to a data storage service by causing multiple servers of the data storage service to collectively receive the plurality of manipulated data chunks

12. The system of claim 11, wherein the instructions further cause the system to:

perform checksum operations on at least a subset of the plurality of manipulated data chunks in parallel; and

cause results of the checksum operations to be used to verify transfer of the subset of the plurality of manipulated data chunks to the data storage service.

13. The system of claim 11, wherein manipulating each data chunk of the plurality of data chunks includes compressing the data chunk.

14. The system of claim 11, wherein at least a subset of the plurality of processors are part of a multi-core central processing unit.

15. The system of claim 11, wherein transferring the plurality of manipulated data chunks to the data storage service is performed in a manner resulting in the data storage service storing the plurality of manipulated data chunks collectively as a second data object.

16. The system of claim 11, wherein:

manipulating the data chunk includes encrypting the data chunk using a corresponding initialization vector;

each data chunk is encrypted using a different corresponding initialization vector; and

transferring the plurality of manipulated data chunks to the data storage service includes transferring each manipulated data chunk with a corresponding initialization vector used to obtain the manipulated data chunk.

17. The system of claim 11, wherein the instructions further cause the system to:

selectively download a proper subset of the plurality of manipulated data chunks without downloading another proper subset of the plurality of manipulated data chunks; and

reconstruct at least a portion of the data object from the selectively downloaded proper subset.

18. A computer-implemented method, comprising:

under the control of one or more computer systems configured with executable instructions,

deconstructing a data object into a first plurality of data chunks;

separately encrypting the first plurality of data chunks such that each subset of a plurality of subsets of the first plurality of data chunks are encrypted using a different initialization vector, the first plurality of data chunks being collectively usable to construct a data object; and

transmitting a second plurality of data chunks to a data storage service using a plurality of parallel channels each corresponding to a separate server of the data storage service, the second plurality of data chunks based at least in part on the first plurality of data chunks.

19. The computer-implemented method of claim 18, wherein each data chunk of the second plurality of data chunks includes an encrypted data chunk from the first plurality of data chunks and an initialization vector used to generate the encrypted data chunk.

20. The computer-implemented method of claim 18, wherein the first plurality of data chunks and the second plurality of data chunks are sets of data chunks containing the same number of members.

21. The computer-implemented method of claim 18, wherein separately encrypting the first plurality of data chunks includes encrypting at least a subset of the first plurality of data chunks in parallel.

22. The computer-implemented method of claim 21, wherein separately encrypting the first plurality of data chunks includes utilizing a non-parallelizable cryptographic primitive.

23. The computer-implemented method of claim 18, further comprising separately downloading in parallel each data chunk of at least a subset of the second plurality of data chunks.