SYSTEM FOR AND METHOD FOR MEDIA UPLOAD MULTITHREADING FOR LARGE FILE UPLOADS

• US 20150163301A1
• Filed: 12/06/2013
• Published: 06/11/2015
• Est. Priority Date: 12/06/2013
• Status: Active Grant

• Current Assignee: Cellco Partnership, Inc. (Verizon Communications Inc.)
• Original Assignee: Cellco Partnership, Inc. (Verizon Communications Inc.)
• Inventors: NARAYANAN, Ajay
• Granted Patent: US 9,197,702 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

Exemplary embodiments provide a system and method for uploading large files to a remote server faster. The system and method divide the file into smaller file chunks and utilize parallel thread execution to prepare and upload the file chunks to the remote server. The uploading may further utilize a plurality of HTTP connections. Once all chunks have been uploaded to the remote server, the chunks may be consolidated into the original file by using content tokens and chunk numbers associated with the uploaded chunks.

Claims

1. A system, comprising:

a processor; and

a memory comprising computer-readable instructions which when executed by the processor cause the processor to perform the steps comprising;

receiving a file having a size property;

determining a maximum upload chunk size for the file;

identifying a total number of file chunks to upload based on the file size property and the maximum upload chunk size;

configuring a maximum number of allowable threads and a maximum number of HTTP connections;

reading a portion of the file into a byte buffer array having a size equal to the maximum upload chunk size;

passing a chunk number, the byte buffer array, and a chunk checksum into one thread associated with the maximum number of allowable threads for upload execution;

uploading the chunk number, the byte buffer array, and the chunk checksum using one HTTP connection of the maximum number of HTTP connections;

receiving, via a network, a content token returned by a receiving server after the byte buffer array, the chunk number, and the chunk checksum have been received at the receiving server;

storing the received content token along with a chunk number as a pair;

determining whether the total number of file chunks have been uploaded; and

when the total number of file chunks have been uploaded, passing all stored content tokens in the order of chunk number and a file checksum to the receiving server for file consolidation.

2. The system of claim 1, wherein the maximum upload chunk size is determined based on at least one of:

a network speed, an available bandwidth, and a system memory availability.

3. The system of claim 1, wherein the maximum number of allowable threads is determined based on at least one of:

a network speed, an available bandwidth, and a system memory availability.

4. The system of claim 1, wherein the maximum number of HTTP connections is determined based on at least one of:

a network speed, an available bandwidth, and a system memory availability.

5. The system of claim 1, wherein the content token is passed to the receiving server in order of chunk number and a file checksum.

6. The system of claim 1, further comprising computing a file checksum based on the file before the configuring step, and wherein the checksum is transmitted to the receiving server to determine whether the file already exists on the receiving server and whether file consolidation generates a complete file of reassembled file chunks.

7. The system of claim 1, wherein the passing of the chunk number, the byte buffer array, and the chunk checksum into one thread associated with the maximum number of allowable threads for upload execution occurs a plurality of times in parallel so that each of the maximum number of allowable threads is utilized in parallel.

8. The system of claim 1, wherein the uploading of the a chunk number, the byte buffer array, and the checksum using one HTTP connection of the maximum number of HTTP connections occurs a plurality of times in parallel so that each of the maximum number of HTTP connections is utilized in parallel.

9. The system of claim 1, wherein the maximum number of allowable threads is double the maximum number of available HTTP connections.

10. The system of claim 9, wherein the maximum number of allowable threads includes at least two sets of threads, where one set is preparing data for upload while the other set is uploading the previously prepared data.

11. A method, comprising:

receiving, on a sending device having at least one processor, a file having a size property;

determining, on the sending device, a maximum upload chunk size for the file;

identifying, on the sending device, a total number of file chunks to upload based on the file size property and the maximum upload chunk size;

configuring, on the sending device, a maximum number of allowable threads and a maximum number of HTTP connections;

reading a portion of the file into a byte buffer array having a size equal to the maximum upload chunk size;

passing a chunk number, the byte buffer array, and a chunk checksum into one thread associated with the maximum number of allowable threads for upload execution;

uploading, to a receiving server, the a chunk number, the byte buffer array, and the chunk checksum using one HTTP connection of the maximum number of HTTP connections;

receiving, via a network, a content token returned by the receiving server after the byte buffer array, chunk number, and chunk checksum have been received at the receiving server;

storing, on the sending device, the received content token along with a chunk number as a pair; and

determining, on the sending device, whether the total number of file chunks have been uploaded, wherein when the total number of file chunks have been uploaded, the processor passes stored content token and chunk number pairs in the order of chunk number and a computed file checksum to the receiving server for file consolidation.

12. The method of claim 11, wherein the maximum upload chunk size is determined based on at least one of:

a network speed, an available bandwidth, and a system memory availability.

13. The method of claim 11, wherein the maximum number of allowable threads is determined based on at least one of:

a network speed, an available bandwidth, and a system memory availability.

14. The method of claim 11, wherein the maximum number of HTTP connections is determined based on at least one of:

a network speed, an available bandwidth, and a system memory availability.

15. The method of claim 11, wherein the content token is passed to the receiving server in order of chunk number and a file checksum.

16. The method of claim 11, further comprising computing a file checksum for the file before the configuring step, and wherein the checksum is transmitted to the receiving server to determine whether the file already exists on the receiving server and whether file consolidation generates a complete file of reassembled file chunks.

17. The method of claim 11, wherein the passing of the chunk number, the byte buffer array, and the chunk checksum into one thread associated with the maximum number of allowable threads for upload execution occurs a plurality of times in parallel so that each of the maximum number of allowable threads is utilized in parallel.

18. The method of claim 11, wherein the uploading of the a chunk number, the byte buffer array, and the checksum using one HTTP connection of the maximum number of HTTP connections occurs a plurality of times in parallel so that each of the maximum number of HTTP connections is utilized in parallel.

19. The method of claim 11, wherein the maximum number of allowable threads is double the maximum number of available HTTP connections.

20. The method of claim 19, wherein the maximum number of allowable threads includes at least two sets of threads, where one set is preparing data for upload while the other set is uploading the previously prepared data.