Systems and Methods to Control Web Scraping

• US 20090288169A1
• Filed: 05/16/2008
• Published: 11/19/2009
• Est. Priority Date: 05/16/2008
• Status: Active Grant

• Current Assignee: Thryv, Inc. (Mudrick Capital Management LP)
• Original Assignee: YellowPages.com LLC (Mudrick Capital Management LP)
• Inventors: MOHS, Bradley Keith, PETTA, Damon Layton
• Granted Patent: US 8,595,847 B2
• Time in Patent Office: Days
• US Class Current: 726/26
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation

Abstract

Systems and methods to control web scraping through a plurality of web servers using real time access statistics are described.

Claims

1. A method to selectively deter web scraping, the method comprising:

storing information on a first database;

providing a web interface to access the information stored in the first database, the web interface including a plurality of web servers capable to independently serve a current web request through retrieving the information from the first database;

in response to the current web request received at one of the web servers, determining identification information indicating a requester of the current web request;

logging the current web request in association with the requester in a second database;

accessing the second database, in response to the current web request, for past web requests of the requester logged in the second database, at least a portion of the past web requests of the requester received via different ones of web servers;

determining whether to block the current web request, to delay the current web request, or to timely reply to the current web request, based on a characteristic of the past web requests logged in the second database; and

in response to a determination to reply to the current web request, providing the information to the requester through one of web servers.

2. The method of claim 1, wherein the characteristic of the past web requests logged in the second database is based on a count of web requests within a predetermined period of time from the current web request.

3. The method of claim 1, wherein the second database provides real time statistics on web requests received at the plurality of web servers;

and the second database is hosted in dynamic random access memories of a plurality of computers that host the plurality of web servers.

4. The method of claim 1, wherein the characteristic of the past web requests logged in the second database is based on an average speed of web requests generated by the requester.

5. The method of claim 1, wherein each of the web servers logs received web requests in the second database;

and each of the web servers accesses real time statistics of web requests received across the plurality of web servers to determine whether the requester is a web scraper.

6. The method of claim 1, further comprising:

determining whether the requester is a web scraper based on whether the characteristic of the past web requests exceeds a threshold.

7. The method of claim 6, further comprising:

determining a category of the current web request; and

selecting the threshold based on the category of the current web request.

8. The method of claim 1, further comprising:

determining whether the requester is a known, friendly scraper based on the identification information;

when the requester is a friendly scraper, determining whether to slow the request based on current workload of the web interface.

9. The method of claim 8, further comprising:

when the requester is a friendly scraper, determining whether to slow the request based on the characteristic of the past web requests.

10. The method of claim 1, further comprising:

determining whether the requester is a known scraper or an end user; and

selecting a format to provide the information based on whether the requester is a known scraper or an end user.

11. The method of claim 1, wherein the accessing of the second database for the past web requests of the requester logged in the second database comprises:

counting web requests of the requester logged in the second database by the web servers within a predetermined period of time prior to the current web request.

12. The method of claim 11, wherein the second database is implemented in a dynamic random access memory of a computer.

13. The method of claim 12, further comprising:

removing entries of logged web requests from the second database when the entries are logged prior to a predetermined period of time.

14. The method of claim 12, further comprising:

purging logged web requests from the second database periodically.

15. The method of claim 1, wherein the identification information indicating the requester of the current web request comprises an Internet address and an identification of user agent.

16. The method of claim 15, further comprising:

extracting a user agent string from the current web request to determine the identification of the user agent.

17. The method of claim 15, further comprising:

maintaining a list of identification information of a plurality of first web scrapers; and

regulating a rate of responses to web requests from the first web scrapers based on real time statistics of web requests logged in the second database.

18. The method of claim 17, further comprising:

maintaining a list of identification information of a plurality of second web scrapers; and

blocking web requests from the second web scrapers.

19. A machine readable media embodying instructions, the instructions causing a data processing system to perform a method, the method comprising:

receiving a current web request at a web interface to access information stored on a first database, the web interface including a plurality of web servers capable to independently serve web requests through retrieving the information from the first database;

in response to the current web request received at one of the web servers, determining identification information indicating a requester of the current web request;

logging the current web request in association with the requester in a second database;

accessing the second database, in response to the current web request, for past web requests of the requester logged in the second database, at least a portion of past web requests of the requester received via different ones of web servers;

determining whether to block the current web request, to delay the current web request, or to timely reply to the current web request, based on a characteristic of the past web requests logged in the second database; and

in response to a determination to reply to the current web request, providing the information to the requester through one of web servers.

20. A data processing system, comprising:

means for receiving a current web request at a web interface to access information stored on a first database, the web interface including a plurality of web servers capable to independently serve web requests through retrieving the information from the first database;

means for determining identification information indicating a requester of the current web request, in response the current web request received at one of the web servers;

means for logging the current web request in association with the requester in a second database, in response to the current web request received at one of the web servers;

means for accessing the second database, in response to the current web request, for past web requests of the requester logged in the second database, at least a portion of the past web requests of the requester received via different ones of web servers;

means for determining whether to block the current web request, to delay the current web request, or to timely reply to the current web request, based on a characteristic of the past web requests logged in the second database; and

means for providing the information to the requester through one of web servers, in response to a determination to reply to the current web request.