Data transfer optimizations

• US 10,187,358 B2
• Filed: 12/03/2013
• Issued: 01/22/2019
• Est. Priority Date: 12/03/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Fahey, Sean Anthony, Hill, Brent James
• Primary Examiner(s): Armouche, Hadi S
• Assistant Examiner(s): Lapian, Alexander R
• Application Number:
  US14/095,898
• Publication Number:
  US 20150156174A1
• Time in Patent Office: 1,876 Days
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/18 using different networks or channels, e.g. using out of band channels cryptographic mechanisms or cryptographic arrangements for key distribution involving distinctive intermediate devices or communication paths H04L9/0827; cryptographic mechanisms or cryptographic arrangements for authentication using a plurality of channels H04L9/3215
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Data transfer between a first computer system and a second computer system utilize parallel servers of the second computer system. A plurality of data chunks collectively comprise a data object. The data chunks may be encrypted and sent over parallel channels to the second computer system, which may be a data storage service of a computing resource service provider. The data object, or a portion thereof, may be downloaded from the data storage system in parallel.

Claims

1. A non-transitory computer-readable storage medium having stored thereon instructions that, upon execution by one or more processors of a computer system, cause the computer system to:

encrypt a plurality of data chunks of a data object with a cryptographic key, wherein the data object is a media file such that;

a first data chunk of the plurality of data chunks is encrypted so as to overlap with encryption of a second data chunk of the plurality of data chunks; and

a subset of the plurality of data chunks is encrypted by the cryptographic key using a first initialization vector that is different than a second initialization vector used to encrypt a different subset of the plurality of data chunks, the first and second initialization vectors forming a set of initialization vectors;

upload the first data chunk of the plurality of data chunks and the first initialization vector of the set of initialization vectors to an identified first server of a data storage service to at least;

store a concatenated data chunk comprising the first data chunk and the first initialization vector;

associate the concatenated data chunk with an identifier corresponding to the data object; and

associate the concatenated data chunk with a second data chunk, the second data chunk stored on a second server of the data storage service of the plurality of data chunks;

retrieve, in connection with a request to retrieve the data object from the data storage service, at least some of the plurality of data chunks in parallel, the request indicating a playback start time different from a beginning time of the media file; and

decrypt at least some of the retrieved plurality of data chunks in parallel, the retrieved data chunks of the plurality of data chunks corresponding to a portion of the media file that includes the playback start time and lacks the beginning time.

2. The non-transitory computer-readable storage medium of claim 1, wherein the instructions that cause the transmitting of the plurality of data chunks and the set of initialization vectors further cause the computer system to transmit the first data chunk as a concatenated sequence of bits that includes a corresponding initialization vector of the set of initialization vectors.

3. The non-transitory computer-readable storage medium of claim 1, wherein the request specifies a range of bytes on at least one of the first server or the second server corresponding to a data chunk of the plurality of data chunks.

4. A non-transitory computer-readable storage medium having stored thereon instructions that, upon execution by one or more processors of a computer system, cause the computer system to:

encrypt a plurality of data chunks of a data object, wherein the data object is a media file such that;

a first data chunk of the plurality of data chunks is encrypted so as to overlap with encryption with a second data chunk of the plurality of data chunks;

a subset of the plurality of data chunks is encrypted using a first initialization vector that is different than a second initialization vector used to encrypt a different subset of the plurality, the first and second initialization vectors forming a set of initialization vectors; and

wherein each of the plurality of data chunks is encrypted using a same cryptographic key;

transmit the first data chunk of the plurality of data chunks and the first initialization vector of the set of initialization vectors to an identified first server of a plurality of servers to at least;

store a first concatenated data generated by combining the first data chunk of the plurality of data chunks with a corresponding first initialization vector of the set of the initialization vectors;

associate the first concatenated data with an identifier corresponding to the data object; and

associate the first concatenated data with the second data chunk, the second data chunk stored on a second server of a data storage service of the plurality of data chunks;

retrieve, in connection with a request to retrieve the data object and from the data storage service, at least some of the plurality of data chunks in parallel, the request indicating a playback start time different from a beginning time of the media file; and

decrypt at least some of the plurality of data chunks in parallel, the retrieved data chunks of the plurality of data chunks corresponding to a portion of the media file that includes the playback start time and lacks the beginning time.

5. The non-transitory computer-readable storage medium of claim 4, wherein the instructions that cause the transmitting of the plurality of data chunks and the set of initialization vectors further cause the computer system to transmit the first data chunk as a concatenated sequence of bits that includes a corresponding initialization vector of the set of initialization vectors.

6. The non-transitory computer-readable storage medium of claim 4, wherein the instructions further cause the computer system to download, from the data storage service, a portion of the plurality of data chunks by, for each encrypted data chunk of the plurality of data chunks, submitting a request to the data storage service that specifies a range of data within the data object that contains the encrypted data chunk.

7. The non-transitory computer-readable storage medium of claim 6, wherein downloading the portion of the plurality of data chunks includes determining, based on a uniform data chunk size of at least some of the data chunks in the portion and a size of the data object, a plurality of ranges each corresponding to a data chunk of the portion.

8. A system, comprising:

memory to store instructions, as a result of being executed by at least one or more processors, cause the system to at least;

process a plurality of data chunks of a media file that collectively comprise a data object by distributing, among the one or more processors, the plurality of data chunks for processing such that at least some data chunks of the plurality of chunks are encrypted in parallel using a cryptographic key and a different respective initialization vector, the respective initialization vectors for the plurality of data chunks forming a set of initialization vectors;

upload a first data chunk of the plurality of processed data chunks and a respective first initialization vector of the set of initialization vectors to an identified first server of a data storage service to at least;

couple, the first data chunk of the plurality of processed data chunks with the respective first initialization vector of the set of initialization vectors to generate a first concatenated data chunk of a plurality of concatenated data chunks;

associate the first concatenated data chunk with an identifier corresponding with the data object; and

associate the first concatenated data chunk with a second data chunk, the second data chunk stored on a second server of the data storage service of the plurality of data chunks; and

retrieve and decrypt in parallel, in response to a request indicating a playback start time different from a beginning time of the media file, at least some of the plurality of data chunks in decrypted form, wherein the retrieved and decrypted data chunks of the plurality of data chunks correspond to a portion of the media file that includes the playback start time and lacks the beginning time.

9. The system of claim 8, wherein the instructions further cause the system to:

perform checksum operations on a subset of the plurality of processed data chunks in parallel; and

cause results of the checksum operations to be used to verify transfer of the subset of the plurality of processed data chunks to the data storage service.

10. The system of claim 8, wherein processing each data chunk of the plurality of data chunks includes compressing the data chunk.

11. The system of claim 8, wherein a subset of the one or more processors are part of a multi-core central processing unit.

12. The system of claim 8, wherein transferring the plurality of processed data chunks to the data storage service is performed in a manner resulting in the data storage service storing the plurality of processed data chunks collectively as a second data object.

13. The system of claim 8, wherein:

transferring each processed data chunk with a corresponding initialization vector includes transmitting first bits that form the processed data chunk and transmitting second bits that form the corresponding initialization vector to a corresponding server.

14. The system of claim 8, wherein the instructions further cause the system to:

selectively download a subset of the plurality of processed data chunks without downloading another subset of the plurality of processed data chunks; and

reconstruct the data object from the selectively downloaded subset.

15. A computer-implemented method, comprising:

processing a data object into a first plurality of data chunks, wherein the data object is a media file;

processing the first plurality of data chunks to generate a second plurality of data chunks such that each data chunk of the first plurality of data chunks is encrypted in parallel using a cryptographic key and a different initialization vector;

uploading the first data chunk of the second plurality of data chunks and a first initialization vector of a set of corresponding initialization vectors to an identified first server of a data storage service to at least;

couple a first data chunk of the second plurality of data chunks with the first initialization vector of the set of initialization vectors to generate a concatenated data chunk base;

store, over at least a partially overlapping period of time, the concatenated data chunk and a second concatenated data chunk on the first server and a second server of the data storage service, respectively;

associate the concatenated data chunk with an identifier corresponding to the data object; and

associate the concatenated data chunk with a data chunk, the data chunk stored on the second server of the data storage service of the second plurality of data chunks; and

retrieve and decrypt, in response to a request indicating a playback start time different from a beginning time of the media file to retrieve the data object in decrypted form, at least some of the plurality of data chunks in parallel, wherein the retrieved and decrypted data chunks of the plurality of data chunks correspond to a portion of the media file that includes the playback start time and lacks the beginning time.

16. The computer-implemented method of claim 15, wherein the transmitting includes transmitting first bits that form the encrypted data chunk and transmitting second bits that form the corresponding initialization vector to a corresponding server.

17. The computer-implemented method of claim 15, wherein the first plurality of data chunks and the second plurality of data chunks include the same number of members.

18. The computer-implemented method of claim 15, wherein processing the first plurality of data chunks includes encrypting at least a subset of the first plurality of data chunks in parallel.

19. The computer-implemented method of claim 18, wherein processing the first plurality of data chunks includes utilizing a non-parallelizable cryptographic primitive.

20. The computer-implemented method of claim 15, further comprising downloading in parallel each data chunk of at least a subset of the second plurality of data chunks.