Parallel random proxy usage for large scale web access

• US 8,010,639 B2
• Filed: 06/05/2009
• Issued: 08/30/2011
• Est. Priority Date: 12/17/2001
• Status: Expired due to Fees

• Current Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Byers, Simon D.
• Primary Examiner(s): LEWIS, CHERYL RENEA
• Application Number:
  US12/479,374
• Publication Number:
  US 20090248843A1
• Time in Patent Office: 816 Days
• Field of Search: 707/713, 707/999.003, 707/802, 709/203, 709/221
• US Class Current: 709/221
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/953 Querying, e.g. by the use of web search engines
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A system and method efficiently and anonymously retrieves large scale Web data through a restricted query interface. A number of proxy servers are utilized to permit parallel access to a target Web server for processing multiple queries simultaneously. Latency in the individual queries is absorbed by the proxy servers. Queries that would otherwise appear structured to the target server are assigned to the proxy server in a random fashion, obscuring the structured nature of the queries. The anonymous nature of the queries made by the proxy servers furthermore conceals the identity of the originating server.

Claims

1. A machine readable medium containing configuration instructions for performing a method for retrieving data accessible by posing a plurality of queries over a network to at least one target server, the method comprising the steps of:

transmitting a first one of the plurality of queries to a first one of a plurality of proxy server services for transmission to one of the at least one target servers;

for each one of the plurality of queries, receiving from its corresponding proxy server service a reply from its corresponding target server, each of said replies comprising data which is at least part of said information; and

constructing a database view of the information using said data received from the proxy server services in reply to said plurality of queries.

2. The machine readable medium of claim 1, wherein the method further comprises the step of:

selecting said plurality of proxy servers services, said selecting comprising the steps of;

searching the network for proxy server services;

for each service found in the searching step, testing the service using test criteria; and

selecting those services meeting the test criteria.

3. The machine readable medium of claim 2, wherein the test criteria include retrieval time.

4. The machine readable medium of claim 2, wherein the test criteria include number of errors in a response to a test query.

5. The machine readable medium of claim 2, wherein the selecting step includes selecting a primary list of proxy services meeting a first set of criteria and a secondary list of proxy services meeting a second set of criteria.

6. The machine readable medium of claim 2, wherein the method further comprises the step of ranking the selected proxy server services according to test criteria, and wherein the transmitting step includes transmitting the query to a best-ranked available one of said services.

7. The machine readable medium of claim 2, wherein said transmitting step comprises transmitting the queries to random ones of the selected proxy server services.

8. The machine readable medium of claim 1, wherein at least one of said plurality of queries is a random or pseudo-random dummy query.

9. The machine readable medium of claim 1, wherein the method further comprises the step of pausing for a substantially unpredictable time interval before performing the transmitting step.

10. The machine readable medium of claim 1, further comprising the step of constructing at least one of said queries based on replies received in response to previous queries.

11. A machine readable medium containing configuration instruction for performing a method for retrieving information accessible by posing a plurality of queries over a network to at least one target server, wherein the plurality of queries are posed through a restricted interface that returns the k data points closest to a query point, the method comprising the steps of:

calculating a maximum radius for a previous query point to a data point returned by the corresponding query;

determining a region within R covered by the previous query based on the corresponding maximum radius;

computing quadtrees of progressively greater levels until a computed quadtree has an uncovered node entirely outside the covered region; and

constructing a subsequent query to contain a query point that is a center of the uncovered quadtree node.

12. The machine readable medium of claim 11, further comprising the step of constructing a database view of the data received from the proxy server.

13. A method for retrieving data accessible by posing a plurality of queries over a network to at least one target server, the method comprising the steps of:

transmitting a first one of the plurality of queries to a first one of a plurality of proxy server services for transmission to one of the at least one target servers;

for each one of the plurality of queries, receiving from its corresponding proxy server service a reply from its corresponding target server, each of said replies comprising data which is at least part of said information; and

constructing a database view of the information using said data received from the proxy server services in reply to said plurality of queries.

14. The method of claim 13, wherein the method further comprises the step of:

selecting said plurality of proxy servers services, said selecting comprising the steps of;

searching the network for proxy server services;

for each service found in the searching step, testing the service using test criteria; and

selecting those services meeting the test criteria.

15. The method of claim 14, wherein the test criteria include retrieval time.

16. The method of claim 14, wherein the test criteria include number of errors in a response to a test query.

17. The method of claim 14, wherein the selecting step includes selecting a primary list of proxy services meeting a first set of criteria and a secondary list of proxy services meeting a second set of criteria.

18. The method of claim 14, further comprising the step of ranking the selected proxy server services according to test criteria, and wherein the transmitting step includes transmitting the query to a best-ranked available one of said services.

19. The method of claim 13, wherein at least one of said plurality of queries is a random or pseudo-random dummy query.

20. The method of claim 13, further comprising the step of pausing for a substantially unpredictable time interval before performing the transmitting step.