Parallel data transfer over multiple channels with data order prioritization

• US 20040172476A1
• Filed: 02/27/2004
• Published: 09/02/2004
• Est. Priority Date: 02/28/2003
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Swarmcast, Inc.
• Inventors: Chapweske, Justin F.
• Granted Patent: US 7,555,559 B2
• Time in Patent Office: Days
• US Class Current: 709/231
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/752 adapting media to network capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/101 based on network conditions
• CPC Class Codes: H04L 67/1021 based on client or server locations
• CPC Class Codes: H04L 67/1023 based on a hash applied to IP addresses or costs
• CPC Class Codes: H04L 67/104 Peer-to-peer [P2P] networks
• CPC Class Codes: H04L 67/1085 involving dynamic management of active down- or uploading connections
• CPC Class Codes: H04L 67/62 Establishing a time schedule for servicing the requests
• CPC Class Codes: H04L 69/14 Multichannel or multilink protocols
• CPC Class Codes: H04L 9/40 Network security protocols

Abstract

A parallel download system is described to download data from a plurality of source servers in parallel. The system may include a source scheduler, a prioritization scheduler, and an I/O dispatcher. The source scheduler generates a download schedule to control which source servers will be downloaded from at a current point in time. The prioritization scheduler tracks the latency and throughput of all channels to the source servers and dynamically adjusts the download schedule to control download performance. The I/O dispatcher produces an output stream to present the downloaded data to one or more external applications or a user.

Claims

1. A parallel download system executing on a client computer to control download of data from a plurality of source servers in parallel, the system comprising:

a source scheduler to generate a download schedule to control which source servers will be downloaded from at a current point in time;

a prioritization scheduler that tracks the latency and throughput of all channels to the source servers and dynamically adjusts the download schedule to control download performance; and

an I/O dispatcher to produce an output stream to present the downloaded data to one or more external applications or a user.

2. The parallel download system of claim 1, wherein the source scheduler keeps track of information about each source server and uses that information to decide which source servers to download from.

3. The parallel download system of claim 1, further comprising a data prioritizer that determines the priority of the data to be scheduled, wherein the data prioritizer specifies an ordering of the data from a highest priority data to a lowest priority data.

4. The parallel download system of claim 3, wherein the prioritization scheduler adjusts the download schedule while maintaining the prioritization the order in which that data will be received.

5. The parallel download system of claim 1, further comprising one or more channel adapters to translate encoded information from a first format used to transfer the information across a channel to a second format desired for a download.

6. The parallel download system of claim 1, further comprising a control interface that allows external applications or users to control the behavior of the parallel download system.

7. The parallel download system of claim 1, further comprising a proportional allocator to determines a proportion of the data that will be transferred from a channel that is currently being scheduled.

8. The parallel download system of claim 1, further comprising a bulk scheduler to determines which bytes of the data will be scheduled.

9. The parallel download system of claim 1, further comprising an advanced scheduler that integrates with the proportional allocator to allow data to be allocated during a different time interval than a current time interval.

10. The parallel download system of claim 9, wherein the advanced scheduler features a sub-interval smoothing component that reduces the burstiness of the availability of high priority data with the proportional allocator.

11. The parallel download system of claim 1, further comprising a constraint scheduler to ensure that the source scheduler does not attempt to retrieve data that a source server cannot provide.

12. The parallel download system of claim 1, further comprising one or more channel receivers to read the data from a respective channel and writes it to an I/O dispatcher.

13. The parallel download system of claim 1, further comprising an integrity verification engine to determine that the desired data is received intact and that none of the channels were providing data that is either corrupt or a different version of the content that is desired.

14. The parallel download system of claim 13, wherein the integrity verification engine utilizes an iterative hash construct, such as a Merkle Hash Tree.

15. The parallel download system of claim 13, further comprising a corruption repair engine to repair data corruption detected by the integrity verification engine.

16. The parallel download system of claim 1, further comprising one or more channel connector to establish a new channel to the scheduled source server.

17. The parallel download system of claim 1, wherein the source scheduler ranks the source servers according to one or more of:

external ranking input received from a user or an administrator or an external ranking service;

a throughput associated with each source server;

a latency associated with each source server;

a number of network hops to the source server from the parallel download system;

a geographical location of each source server relative to the parallel download system; and

a channel cost associated with the channel from the parallel download system to each source server.

18. The parallel download system of claim 3, wherein the prioritization scheduler schedules ranges of bytes to be downloaded.