Method and system for characterising a web site by sampling

• US 8,136,029 B2
• Filed: 09/26/2008
• Issued: 03/13/2012
• Est. Priority Date: 07/25/2008
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Jagdale, Prajakta, Hoffman, Billy
• Primary Examiner(s): Hong, Stephen
• Assistant Examiner(s): Patel, Manglesh
• Application Number:
  US12/239,461
• Publication Number:
  US 20100023850A1
• Time in Patent Office: 1,264 Days
• Field of Search: 715/234, 715/200, 715/206
• US Class Current: 715/206
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

A method of characterising a web site by sampling, the method comprising, the repeated steps of: accessing a URL; receiving a web page; analysing the URL and received webpage and recording characteristics thereof; identifying links within the received web page; grouping links within the received web-page based on proximity; and selecting one of the selected links for subsequent access based on the grouping. The method can be applied in a web application assessment tool.

Claims

1. A method of characterising a web site by sampling, the method comprising:

accessing a URL;

receiving a web page;

analysing the URL and received webpage and recording characteristics thereof;

identifying links within the received web page;

grouping links within the received web-page based on proximity, wherein proximity of links with a page is determined using the number of characters in the web page content that are not part of tag data between the end of a first tag and the beginning of a second tag; and

selecting one of the selected links for subsequent access based on the grouping.

2. A method as claimed in claim 1 comprising scoring each link within the received web page so that a stored score variable corresponds, at least in part, to the frequency of occurrence of the link in the received web pages and wherein the selecting is further based on the stored score variable.

3. A method as claimed in claim 1 comprising excluding links for selection based on the presence of keywords in data associated with the link.

4. A method as claimed in claim 3 wherein the keywords are arranged in a white list comprising a plurality of words that indicate that the link may be of significance to the overall structure of the site and a black list comprising a plurality words that indicate the link is likely not relevant to the structure of the site.

5. A method as claimed in claim 1 comprising repeating the steps.

6. A method as claimed in claim 1 wherein the stored characteristics are least one of filetype, authentication requirements, hostnames, ports, query parameters, forms.

7. A system for characterising a web site by sampling, the system comprising:

a computer that executes a software program and performs each of;

accessing a URL;

receiving a web page;

analysing the URL and received webpage and recording characteristics thereof;

identifying links within the received web page;

grouping links within the received web-page based on proximity, wherein proximity of links with a page is determined using the number of characters in the web page content that are not part of tag data between the end of a first tag and the beginning of a second tag; and

selecting one of the selected links for subsequent access based on the grouping.

8. A system as claimed in claim 7 wherein the crawl function scores each link within the received web page so that a stored score variable corresponds, at least in part, to the frequency of occurrence of the link in the received web pages and wherein the selecting is further based on the stored score variable.

9. A system as claimed in claim 7 wherein the crawl function excludes links for selection based on the presence of keywords in data associated with the link.

10. A system as claimed in claim 9 wherein the keywords are arranged in a white list comprising a plurality of words that indicate that the link may be of significance to the overall structure of the site and a black list comprising a plurality words that indicate the link is likely not relevant to the structure of the site.

11. A web application assessment tool comprising a system as claimed in claim 7 and a settings function, wherein the recorded characteristics are used to adjust the settings.

12. A web application assessment tool, comprising:

a computer that executes a software program and performs each of;

accessing a URL within a target website;

receiving a web page;

analysing the URL and received webpage and recording characteristics thereof;

identifying links within the received web page;

grouping links within the received web-page based on the number of characters in the web page content that are not part of tag data between the end of a first tag and the beginning of a second tag;

scoring each link within the received web page so that a stored score variable corresponds, at least in part, to the frequency of occurrence of the link in the received web pages;

selecting one of the selected links for subsequent access based on the grouping and on the stored score variable;

excluding links for selection based on the presence of keywords in data associated with the link, wherein the keywords are arranged in a white list comprising a plurality of words that indicate that the link may be of significance to the overall structure of the site and a black list comprising a plurality words that indicate the link is likely not relevant to the structure of the site;

a settings function, wherein the recorded characteristics are used to adjust the settings; and

a crawl and attack function for vulnerability scanning the target website using the adjusted settings.

13. A web application assessment tool as claimed in claim 12 for storing a software program implementing the precrawl function, the settings function and the crawl and attack function.