Method for downloading high-volumes of content from the internet without adversely effecting the source of the content or being detected

• US 7,516,194 B1
• Filed: 05/21/2003
• Issued: 04/07/2009
• Est. Priority Date: 05/21/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: O'Brien, Rick, Lamkins, Nick, Dirksen, James
• Primary Examiner(s): Kang; Paul H
• Assistant Examiner(s): Ibrahim; Mohamed
• Application Number:
  US10/443,110
• Time in Patent Office: 2,148 Days
• Field of Search: 709217-219
• US Class Current: 709/218
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/567 Integrating service provisioning from a plurality of service providers
• CPC Class Codes: H04L 69/40 for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection management of faults, events, alarms or notifications in data switching networks H04L41/06

Abstract

A method and system for downloading high volumes of content from the Internet without adversely affecting the source of the content or being detected. A director server processes data received from a data source to generate a list of addresses for content to download. The director server assigns a portion of the list of addresses to one or more puller servers. Puller servers initiate requests to download content that corresponds to their assigned list of addresses. A proxy gateway server receives the download request for content, and downloads the content via various Internet Protocol addresses.

Claims

1. A system for downloading a plurality of documents from a plurality of content servers, said content servers being linked to a plurality of routers that each have a different network address, said system comprising:

a plurality of pullers;

a director for;

creating a list of URLs of the plurality of documents to be downloaded from the plurality of content servers, each of the plurality of said documents being identified by a different URL; and

assigning a portion of the list of URLs to each of the pullers such that each portion assigned to a particular puller includes all documents to be retrieved from a single content server wherein no two pullers initiate requests to adjacent URLs, wherein adjacent URLs identify documents located on the same content server;

wherein each of the plurality of pullers is responsive to the director for;

receiving the assigned portion of the list of URLs;

queuing requests to retrieve documents identified by the received portion of the list of URLs wherein the requests having different URLs are queued by the puller;

determining if the URL of a first queued request is adjacent to the URL of a document being currently downloaded;

if the URL of the first queued request is adjacent to the URL of a document being currently downloaded, waiting until the currently downloading document has been received before initiating the first queued request to avoid overlapping requests to the content server;

if the URL of the queued request is not adjacent to the URL of a document being currently downloaded, initiating the first queued request; and

a proxy gateway responsive to each of the pullers for receiving the initiated requests to retrieve documents, and for retrieving documents corresponding to the list of URL from the content servers via the routers.

2. The system of claim 1, wherein the plurality of routers are linked to the content servers via an external communication network.

3. The system of claim 2, wherein the proxy gateway includes:

an internal interface for interfacing with an internal communication network and for receiving the requests to retrieve documents; and

a plurality of external interfaces for interfacing with the external communication network via the plurality of routers.

4. The system of claim 3, wherein the internal communication network is a local area network (LAN), a wide area network (WAN), or the Internet.

5. The system of claim 3, wherein the external communication network is the Internet.

6. The system of claim 3, wherein the proxy gateway appears to initiate the requests to retrieve documents via one or more of the external interfaces, and wherein each external interface is identified as having the network address of the router to which it is interfaced.

7. The system of claim 6, wherein the proxy gateway is responsive to the received requests for executing a first proxy application to assign one or more of the requests to retrieve documents to each of the plurality of external interfaces, and wherein the proxy gateway is responsive to the one or more assigned requests for executing a second proxy application to retrieve one or more documents corresponding to the one or more assigned requests.

8. The system of claim 1, wherein each of the plurality of routers is assigned a network address by an Internet service provider.

9. The system of claim 1, wherein each of the plurality of pullers is further responsive to the director for examining content within the retrieved documents to identify additional URLs which are added to the list of URLs, wherein the additional URLs identify additional documents to be retrieved.

10. The system of claim 1, further comprising a feature extraction tool for extracting features such as links and URLs to identify additional URLs of documents that are referenced in an HTML file.

11. The system of claim 1, wherein the director creates the list of URLs by cataloging, queuing, sorting, and prioritizing the list of URLs that correspond to the plurality of documents.

12. A method for retrieving a plurality of documents from a content server, comprising:

creating a list of URLs corresponding to the plurality of documents to be downloaded, each of said documents being identified by a different URL;

queuing requests to retrieve documents corresponding to the list of URLs from the content server wherein requests to the content server having different URLs are queued;

determining if the URL of a first queued request is adjacent to the URL of a document being currently downloaded, wherein adjacent URLs identify documents located on the same content server;

if the URL of the first queued request is adjacent to the URL of a document being currently downloaded, waiting until the current document has been downloaded by the content server before initiating one of the queued requests to avoid overlapping requests to the content server;

if the URL of the queued request is not adjacent to the URL of a document being currently downloaded, then initiating one of the queued requests;

assigning a portion of documents corresponding to the initiated requests to be retrieved via each of a plurality of network addresses; and

retrieving the requested documents from the content server via the plurality of network addresses, wherein each of the network addresses appears to indicate a location from which the document requests were initiated.

13. The method of claim 12, wherein the list of URLs corresponding to the plurality of documents is provided by a user, an affiliate server, or links within retrieved documents.

14. The method of claim 12, wherein creating a list of URLs includes cataloging, queuing, sorting, and prioritizing a list of URLs corresponding to the plurality of documents to be retrieved.

15. The method of claim 12, wherein retrieving the requested documents includes retrieving a portion of the requested documents from the content server via each of the plurality of network addresses.

16. The method of claim 12, further comprising:

examining content within retrieved documents;

identifying additional URLs for documents from the examined content; and

adding the identified additional URLs the to the list of URLs to be retrieved.

17. A computer readable storage medium having computer executable instructions for downloading a plurality of documents from a content server in response to download requests, comprising:

retrieving instructions for;

retrieving download requests for documents via an internal interface, each of said documents being identified by a different URL;

queuing download requests to retrieve documents from the content server wherein requests to the content server having different URLs are queued; and

determining if the URL of a first queued request is adjacent to the URL of a document being currently downloaded, wherein adjacent URLs identify documents located on the same content server;

if the URL of the first queued request is adjacent to the URL of a document being currently downloaded, waiting until the current document has been downloaded by the content server before initiating one of the queued requests to avoid overlapping requests to the content server; and

if the URL of the first queued request is adjacent to the URL of a document being currently downloaded, initiating one of the queued requests;

distributing instructions for distributing the initiated download requests among a plurality of external interfaces, and wherein each external interface has a different IP address for communicating with the content server; and

downloading instructions for downloading documents corresponding to the initiated download requests from the content server via the plurality of external interfaces.

18. The computer readable storage medium of claim 17, wherein the distributing instructions assign one or more of the requests to retrieve documents to each of the plurality external interfaces.

19. The computer readable storage medium of claim 17, wherein the downloading instructions modify information contained in the assigned request to identify the request as being initiated via the external interface to which the request is assigned.

20. The computer readable storage medium of claim 17, wherein the download requests for documents are retrieved via a local area network (LAN), wide area network (WAN), or the Internet.

21. A system for downloading a plurality of documents from a plurality of content servers, said content servers being linked to a plurality of routers that each have a different network address, said system comprising:

a plurality of pullers;

a director for;

creating a list of document addresses corresponding to the plurality of documents to be downloaded, each of the plurality of said documents being identified by a different document address; and

assigning a portion of the list of document addresses to each of the pullers such that each portion assigned to a particular puller includes all documents to be retrieved from one content server wherein no two pullers are assigned document addresses to be retrieved from the same content server;

wherein a first puller is responsive to the director for;

receiving a first portion of the list of document addresses;

queuing a first request to retrieve a first document based on a first document address and queuing a second request to retrieve a second document based on a second document address, said first and second document address included in the list of document addresses;

initiating the first queued request to retrieve the first document from a first content server based on the first document address;

based on the second document address of the second queued request, determining the content server identified by the second document address;

if the determined content server is the first content server and the first content server is in the process of retrieving the first document, waiting until the first document has been received by the first content server before initiating the second queued request to avoid overlapping requests to the first content server;

if the determined content server is not the first content server or the first content server is not in the process of retrieving the first document, initiating the second queued request to retrieve the second document from the determined content server; and

a proxy gateway responsive to each of the pullers for receiving the initiated requests to retrieve documents, and for retrieving documents corresponding to the list of document addresses from the content servers via the routers.

22. The system of claim 21, wherein a first puller is responsive to the director for:

receiving a second portion of the list of document addresses;

queuing a third request to retrieve a third document based on a third document address and queuing a fourth request to retrieve a fourth document based on a fourth document address, said third and fourth document address included in the second portion of the list of document addresses;

initiating the third queued request to retrieve the third document from a third content server based on the third document address;

based on the fourth document address of the fourth queued request, determining the content server identified by the fourth document address;

if the determined content server is the third content server and the third content server is in the process of retrieving the third document, waiting until the third document has been received by the third content server before initiating the fourth queued request to avoid overlapping requests to the third content server;

if the determined content server is not the third content server or the third content server is not in the process of retrieving the third document, initiating the fourth queued request to retrieve the fourth document from the determined content server.