System and method of analyzing web content

• US 20080010368A1
• Filed: 07/10/2006
• Published: 01/10/2008
• Est. Priority Date: 07/10/2006
• Status: Active Grant

• Current Assignee: Forcepoint LLC
• Original Assignee: Forcepoint LLC
• Inventors: Baddour, Victor Louise, Hubbard, Dan, Verenini, Nicholas Joseph
• Granted Patent: US 8,020,206 B2
• Time in Patent Office: Days
• US Class Current: 709/223
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002

Abstract

A system and method are provided for identifying inappropriate content in websites on a network. Unrecognized uniform resource locators (URLs) or other web content are accessed by workstations and are identified as possibly having malicious content. The URLs or web content may be preprocessed within a gateway server module or some other software module to collect additional information related to the URLs. The URLs may be scanned for known attack signatures, and if any are found, they may be tagged as candidate URLs in need of further analysis by a classification module.

Claims

1. A computer-implemented method of identifying inappropriate web content, the method comprising:

receiving a request for web content;

comparing the request to data in a database;

sending the request to a collection module if the request is not in the database;

collecting, by the collection module, data related to the request; and

determining a candidate status for the request based on the collected data.

2. The computer-implemented method of claim 1, wherein the request is an application request.

3. The computer-implemented method of claim 1, wherein the request is a request for a URL.

4. The computer-implemented method of claim 1, wherein the database is a URL/content database.

5. The computer-implemented method of claim 1, wherein the web content is content accessible through a URL.

6. The computer-implemented method of claim 1, wherein determining whether to make the request a candidate request comprises:

preprocessing the data related to the request; and

tagging the request as a candidate request at least partially based on the preprocessing.

7. The computer-implemented method of claim 6, wherein the preprocessing includes scanning the data for known unsafe data elements.

8. The computer-implemented method of claim 7, wherein the preprocessing indicates that the data related to the request contains a known unsafe data element.

9. The computer-implemented method claim 7, further comprising tagging the request as a potential candidate request if the data related to the request does not contain a known unsafe data element.

10. The computer-implemented method of claim 9, further comprising:

configuring a data mining module to select candidate requests from the potential candidate request; and

inputting the potential candidate requests into the data mining module.

11. The computer-implemented method of claim 10, wherein configuring the data mining module includes defining a characteristic indicative of a targeted attribute, and configuring the data mining module to identify requests having the attribute.

12. The computer-implemented method of claim 11, wherein the attribute is a set of at least one of keywords, regular expressions, or operands.

13. The computer-implemented method of claim 11, wherein the attribute is a type of HTTP request header data.

14. The computer-implemented method of claim 11, wherein the HTTP request header data includes a content-type.

15. A system for selecting candidate URLs from a set of uncategorized URLs, the system comprising:

a database storing the uncategorized URLs;

a collection system configured to collect information related to the uncategorized URLs; and

a data mining module configured to identify uncategorized URLs having a characteristic indicative of targeted content.

16. The system of claim 15, wherein the targeted content is harmful content.

17. The system of claim 15, further comprising a collection module configured to collect additional data about the uncategorized URLs, the additional data being used to identify the targeted content.

18. The system of claim 17, wherein the collection module includes a webcrawling module.

19. The system of claim 18, wherein the collection module includes a keyword data module.

20. The system of claim 19, further comprising a priority module configured to prioritize candidate URLs for categorization based on a potential danger of the URLs.

21. A computer-implemented method of collecting data about URLs, the method comprising:

providing a data mining module with a configuration plug-in, the data mining module having a plurality of dispatchers configured to operate independently of each other;

receiving URL data into the data mining module for analysis;

separating the URL data into work units, each work unit comprising a URL;

determining whether one of the plurality of dispatchers is available for receiving a work unit;

sending one of the work units to one of the dispatchers if available; and

processing the sent work unit based on data provided by the configuration plug-in.

22. The computer-implemented method of claim 21, wherein the dispatchers each comprises a separate process within a computer memory.

23. The method of claim 22, wherein the configuration plug-in provides an instruction to the available dispatcher, the instruction causing the available dispatchers to visit a web page related to the processed sent work unit and to store the data associated with the processed sent work unit in a database.

24. The method of claim 23, wherein if the data associated with the processed sent work unit includes a URL string, adding the URL string to the URL data.

25. The method of claim 21, wherein the determining whether one of the plurality of dispatchers is available for receiving a work unit comprises polling the dispatchers.

26. A system for collecting data about URLs, the system comprising:

a database for storing information about URLs;

a pool of dispatchers, the dispatchers comprising asynchronous system processes each configured to receive URL data input and perform actions on the data; and

a driver module configured to monitor the pool of dispatchers for available dispatchers, and send part of the URL data input to the available dispatchers.

27. The system of claim 26, wherein the dispatchers are further configured to visit a URL indicated by the received URL data input and to download information associated with the URL data input to the database.

28. A system for identifying candidate URLs from a set of uncategorized URLs, the system comprising:

means for storing the uncategorized URLs;

means for collecting information related to the uncategorized URLs; and

means for identifying the uncategorized URLs having a characteristic indicative of targeted content.