Methods and systems for scanning and monitoring content on a network

• US 20060106866A1
• Filed: 10/29/2004
• Published: 05/18/2006
• Est. Priority Date: 10/29/2004
• Status: Active Grant

• Current Assignee: Sysxnet Ltd.
• Original Assignee: Trustwave Corporation (Singapore Telecommunications Limited)
• Inventors: Green, Kenneth, Smith, Phillip, Patanella, Joseph
• Granted Patent: US 8,683,031 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 63/123 received data contents, e.g. message integrity
• CPC Class Codes: H04L 63/1408 by monitoring network traffic monitoring network traffic per se H04L43/00
• CPC Class Codes: H04L 63/1483 service impersonation, e.g. phishing, pharming or web spoofing detection of rogue wireless access points H04W12/12
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

The invention addressed methods and systems of scanning and monitoring network locations and resources such as websites and webpages. The method scans the webpages for content and compliance with predefined standards, and reports the results in a non-technical format. The method can also scan and monitor any website on the Internet, categorize its content, and validate its compliance with application, presentation or content standards, or any combination of the above.

Claims

1. A method for scanning and monitoring content on a computer network, the method comprising:

scanning the network to identify network resources on which the relevant content appears and the location of said resources on the network;

scanning each of the identified network locations to determine any network address information, said network address information identifying the computer or computer system related to that network location;

resolving each of said identified network locations by network address information into one or more network addresses;

profiling said network resources by classifying the content available at said resource;

acquiring said relevant content, said acquiring including copying said network resource and said content to another computer system;

analyzing said network resource by breaking up the content at that network resource into one or more constituent elements;

identifying and analyzing any links on a network resource to any other network resource and categorizing said links into local, neighbor, or remote links based on their relationship with the current resource being analyzed;

identifying broken links, in which the network resource linked to is not available;

keyword scanning of content on said network resource for predetermined keywords or phrases and determining whether said keywords are present on the network resource being analyzed;

analyzing content for language patterns to obtain instances of information such as street addresses, phone numbers, email addresses and other pattern-defined items;

fingerprinting content on said network resource to obtain a quantitative fingerprint for at least one of said constituent elements of said content; and

fingerprinting said network location to obtain a single quantitative measurement for all of said network resources and content at said network location.

2. The method of claim 1, wherein said network is the Internet, wherein said network location is a website, and said network resource is a web page.

3. The method of claim 1, further comprising detecting whether said network location is compromised as a result of hardware malfunction, software error, virus, or network intrusion.

4. The method of claim 1, wherein said constituent elements comprise content type, authentication requirements, page title, hidden text, embedded text such as meta-keywords and meta-descriptors;

visible text, links to other network resources, and embedded resources such as images, media files and non-text searchable documents.

5. The method of claim 1, wherein said network is the Internet and wherein said content includes webpages, elements of webpages, visible text displayable in a browser used to display webpages on the Internet.

6. The method of claim 1 wherein said network comprises an intranet network which is not publicly accessible.

7. The method of claim 1, wherein scanning said network comprises searching the Internet by keyword to obtain a list of websites.

8. The method of claim 1 wherein said scanning each of the identified network locations further comprises scanning one or more network addresses for computer systems running Internet web servers, wherein the extent of said site scan can be configured based on the number of HTTP ports or TCP ports to be scanned.

9. The method of claim 1 wherein said scanning each of the identified network locations further comprises identifying the IP addresses of each of said web servers or web services based on a domain name search resolution methodology, and wherein scanning each of the identified network locations also identifies multiple IP addresses that are used for a single website in a load balancing scheme.

10. The method of claim 1 wherein said profiling further comprises obtaining specific information about the network resource being profiled, said specific information comprising:

title of the network resource;

initial redirects such as HTTP and meta-refresh redirects, X.509 certificate information; and

the vendor information and technical information of the computer systems on which said network resource is located.

11. The method of claim 10, further comprising classifying said network resource being profiled into one or more predetermined categories based on keyword scanning performed during said profiling.

12. The method of claim 1 wherein acquiring of said content includes copying to a computer system all accessible content including but not limited to HTML, style sheets, XML, images, non-text documents, and any other data files, media files or media resources that are part of the network location being profiled.

13. The method of claim 1, wherein said keyword scanning further comprises scanning said network content for forbidden content, wherein said forbidden content is content that is not to be present on any network resources.

14. The method of claim 1 wherein said keyword scanning further comprises content validation, wherein said content validation verifies that predetermined content that is required on the network resources in a predetermined manner is present.

15. The method of claim 3, wherein said detection of said compromised network location further comprises determining whether a network topology of said network location has changed.

16. The method of claim 1, wherein said fingerprinting content further comprises validating that the contents of each media file are accurately reflected by its filename extension.

17. The method of claim 16, wherein said fingerprinting content comprises determining a checksum for said content.

18. The method of claim 1, further comprising monitoring said network location for changes to network resources or content, wherein said monitoring comprises periodically obtaining a new fingerprint for the network location and comparing this new fingerprint with a previously stored fingerprint for the same network location.

19. A method for scanning and monitoring a website on the Internet, comprising:

scanning web pages at said site;

summarizing of the website by identifying the content, name, title, number of predetermined keyword fits and categories of at least one webpage on said site; and

summarizing technical information about said site, including identifying the vendors information and IP addresses of any web servers on which said website runs.

20. The method of claim 19, further comprising:

providing details of the predetermined keyword hits on a webpage such as the information on said webpage surrounding said predetermined keyword hits, the authentication requirements for the webpage, the embedded links on the webpage, and any images and any other media files on the webpage.

21. The method of claim 19, further comprising providing a list of analytic events relating to the website including information regarding compromised webpages or changes in said network topology.

22. The method of claim 19, further comprising providing a list of webpage errors including broken links, spelling errors, images and media files that have not been validated, HTML errors and any missing content that is required to be present.

23. The method for scanning and monitoring one or more websites associated with an enterprise, comprising:

detecting the presence of forbidden content on any of said websites;

validating the presence of required content on one or more webpages on one or more of said websites; and

inventorying and profiling any pages on any of said websites, wherein said inventorying comprises archiving and storing all of the webpages on all of the websites associated with the organization,

wherein said forbidden content and said required content are determined prior to the initiation of the scanning of said websites.

24. The method of claim 23, further comprising:

obtaining a list of domain names and webpages associated with said enterprise;

obtaining a list of network blocks to scan for said webservers; and

obtaining a list of keywords which are used to search the Internet, wherein the obtaining of said domain names, network blocks and said list of search results determined a the website that are associated with said enterprise.

25. A system for scanning content on a network comprising:

a tasking interface;

a tasking database;

traffic acquisition modules;

local data storage;

analysis infrastructure modules;

an event database; and

a reporting analysis interface;

wherein said tasking interface accepting input relating to said scanning,

wherein the tasking database contains information regarding the scheduling and configuration of the traffic acquisition modules to acquire data traffic from the network,

wherein the data acquired by the traffic acquisition modules is stored in the raw data storage,

wherein the analysis infrastructure modules analyze content stored in the raw data storage,

wherein the output of the content analysis infrastructure modules is stored in the event database with information regarding action to be taken based on content that is detected.

26. The content scanning system of claim 25, wherein said traffic acquisition modules include one or more of a service proxy module, a site discovery module, a website acquisition module, an FTP site acquisition module, a network monitor module, and a real-time monitor and countermeasures module.

27. The content scanning system of claim 26, wherein one or more of the traffic acquisition modules operate in and active mode in which the acquisition modules acquire data from a location external to said network on a predetermined basis.

28. The content scanning system of claim 25, wherein one or more of the traffic acquisition modules operate in a passive mode in which traffic is scanned as it passes across the network without interrupting the flow of said traffic.

29. The content scanning system of claim 25, wherein one or more of the traffic acquisition modules operate in a proxy mode in which traffic passing through the network is interrupted by the acquisition module and is prevented from passing to or from the network until said content is analyzed.

30. The content scanning system of claim 25, wherein the analysis infrastructure modules comprise one or more of the following:

website analyzer to analyze websites, a text parser to extract text data, a link analyzer to verify the presence of a network resource identified by a network link, a media analyzer to analyze a media type of data contained with a file identified as a media file, an attachment decoder, an image analyzer to analyze image data contained with a file identified as an image file and a form analyzer.

31. The content scanning system of claim 25, wherein the analysis infrastructure modules extract data from the traffic stream to a normalized traffic file regardless of the protocol under which that data is passed across the network, the application used to generate or to receive the data, or the manner in which said data is encapsulated, wherein all of said extracted data is presented in an equivalent format in the normalized traffic file.

32. The content scanning system of claim 25, further comprising a content scanner to categorize and search of the normalized traffic file based upon predetermined key words, search terms or categories.

33. The content scanning system of claim 25, wherein the output of the content scanner is passed to the event database where said output is stored along with action to be taken based upon said output.

34. A method of scanning a target network resource on a network, comprising:

determining a network address of the network resource;

scanning the network for network resources related to said target network resource; and

analyzing content of any network resource found at an address related to the network address of the target network resource.

35. The method of scanning a target network resource of claim 34, wherein said network is the Internet, said network resource is a website, and said network address includes an IP address.

36. The method of scanning a target network resource of claim 35, wherein said scanning the network for network resources related to said target network resource comprises scanning IP addresses within a range of addresses.

37. The method of scanning a target network resource of claim 35, wherein said scanning the network for network resources related to said target network resource comprises:

identifying a datacenter having a web server that hosts the target website;

obtaining the IP address of said datacenter;

and scanning IP addresses above and below the IP address of said datacenter.

38. The method of scanning a target network resource of claim 35, wherein said scanning the network for network resources related to said target network resource comprises identifying the IP address of the web server on which the target website is hosted and scanning the ports of that IP address.

39. The method of scanning a target network resource of claim 34, wherein said scanning the network for network resources related to said target network resource further comprises:

identifying business information about the target network resource;

looking up at least a portion of said business information in a directory, said directory relating business information to network address information; and

scanning a network resource located using said network resource information.

40. The method of scanning a target network resource of claim 39, wherein said business information includes mailing address information, and wherein said directory is the WHOIS database.