Method And System For Characterising A Web Site By Sampling

• US 20100023850A1
• Filed: 09/26/2008
• Published: 01/28/2010
• Est. Priority Date: 07/25/2008
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Hoffman, Billy, Jagdale, Prajakta
• Granted Patent: US 8,136,029 B2
• Time in Patent Office: Days
• US Class Current: 715/206
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method of characterising a web site by sampling, the method comprising, the repeated steps of: accessing a URL; receiving a web page; analysing the URL and received webpage and recording characteristics thereof; identifying links within the received web page; grouping links within the received web-page based on proximity; and selecting one of the selected links for subsequent access based on the grouping. The method can be applied in a web application assessment tool.

Claims

1. A method of characterising a web site by sampling, the method comprising, the repeated steps of:

accessing a URL;

receiving a web page;

analysing the URL and received webpage and recording characteristics thereof;

identifying links within the received web page;

grouping links within the received web-page based on proximity; and

selecting one of the selected links for subsequent access based on the grouping.

2. A method as claimed in claim 1 comprising scoring each link within the received web page so that a stored score variable corresponds, at least in part, to the frequency of occurrence of the link in the received web pages and wherein the selecting is further based on the stored score variable.

3. A method as claimed in claim 1 comprising excluding links for selection based on the presence of keywords in data associated with the link.

4. A method as claimed in claim 3 wherein the keywords are arranged in a white list comprising a plurality of words that indicate that the link may be of significance to the overall structure of the site and a black list comprising a plurality words that indicate the link is likely not relevant to the structure of the site.

5. A method as claimed in claim 1 comprising repeating the steps a limited number of times.

6. A method as claimed in claim 1 wherein the stored characteristics are least one of filetype, authentication requirements, hostnames, ports, query parameters, forms.

7. A method as claimed in claim 1 wherein proximity of links with a page is determined as the number of characters in the web page content that are not part of tag data between the end of a first tag and the beginning of a second tag.

8. A system for characterising a web site by sampling, the system comprising a crawl function for repeatedly:

accessing a URL;

receiving a web page;

analysing the URL and received webpage and recording characteristics thereof;

identifying links within the received web page;

grouping links within the received web-page based on proximity; and

selecting one of the selected links for subsequent access based on the grouping.

9. A system as claimed in claim 8 wherein the crawl function scores each link within the received web page so that a stored score variable corresponds, at least in part, to the frequency of occurrence of the link in the received web pages and wherein the selecting is further based on the stored score variable.

10. A system as claimed in claim 8 wherein the crawl function excludes links for selection based on the presence of keywords in data associated with the link.

11. A system as claimed in claim 10 wherein the keywords are arranged in a white list comprising a plurality of words that indicate that the link may be of significance to the overall structure of the site and a black list comprising a plurality words that indicate the link is likely not relevant to the structure of the site.

12. A system as claimed in claim 8 wherein proximity of links with a page is determined as the number of characters in the web page content that are not part of tag data between the end of a first tag and the beginning of a second tag.

13. A web application assessment tool comprising a system as claimed in claim 8 and a settings function, wherein the recorded characteristics are used to adjust the settings.

14. A web application assessment tool comprising a precrawl function for repeatedly:

accessing a URL within a target website;

receiving a web page;

analysing the URL and received webpage and recording characteristics thereof;

identifying links within the received web page;

grouping links within the received web-page based on the number of characters in the web page content that are not part of tag data between the end of a first tag and the beginning of a second tag;

scoring each link within the received web page so that a stored score variable corresponds, at least in part, to the frequency of occurrence of the link in the received web pages;

selecting one of the selected links for subsequent access based on the grouping and on the stored score variable;

excluding links for selection based on the presence of keywords in data associated with the link, wherein the keywords are arranged in a white list comprising a plurality of words that indicate that the link may be of significance to the overall structure of the site and a black list comprising a plurality words that indicate the link is likely not relevant to the structure of the site;

a settings function, wherein the recorded characteristics are used to adjust the settings;

and a crawl and attack function for vulnerability scanning the target website using the adjusted settings.

15. A web application assessment tool as claimed in claim 14 in the form of computer-readable media for storing a software program implementing the precrawl function, the settings function and the crawl and attack function.