Integrated crawling and auditing of web applications and web content

• US 7,765,597 B2
• Filed: 08/01/2006
• Issued: 07/27/2010
• Est. Priority Date: 02/11/2004
• Status: Expired due to Fees

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Sullivan, Bryan, Millar, Steve, Kelly, Raymond, Tillery, David, Sima, Caleb, Sullivan, Gerald
• Primary Examiner(s): Barron, Jr.; Gilberto
• Assistant Examiner(s): ARMOUCHE, HADI S
• Application Number:
  US11/461,766
• Publication Number:
  US 20070061877A1
• Time in Patent Office: 1,456 Days
• Field of Search: 726/12, 726 22- 25, 726/5, 725/25, 713/164, 713/188, 707/709, 707/710, 707/781, 712/201
• US Class Current: 726/25
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

A vulnerability assessment tool that is operative to analyze web sites by simultaneously operating a crawling process and an audit process. Once the crawling process is invoked, the results are provided to the audit process. The audit process, rather than waiting until the crawl process is completed, simultaneously audits the web site based on the already provided crawl results. The results of the audit are also fed back to the crawl process to further enhance the crawl.

Claims

1. A method for performing a vulnerability assessment of a target object, the method comprising the steps of:

identifying a target object to assess;

initiating a crawling process on the identified object;

storing the results of the crawling process into a queue in a memory storage device;

initiating an audit process while the crawling process is still in operation and additional results of the crawling process are being stored into the memory storage device;

extracting one or more stored results of the crawling process from the memory storage device and providing them to the audit process;

storing the results of the audit process into the memory storage device;

extracting the stored results of the audit process from the memory storage device and providing them to the crawling process, while the crawling process is still in operation, for further crawling; and

allocating processing power to the co-operating crawling process and the audit process based on the number of results existing in the queue.

2. The method of claim 1, wherein the target object is a web site and the step of identifying the target object comprises identifying a URL for the web site and the step of initiating a crawling process further comprises providing the URL to the crawling process.

3. The method of claim 2, wherein the step of initiating a crawling process further comprises examining the HTML and objects associated with the URL.

4. The method of claim 1, wherein the target object is a web application accessible at a URL, and the step of initiating a crawling process involves examining the web application.

5. The method of claim 1, wherein the step of initiating a crawling process comprises searching the target object for links, sessions and files.

6. The method of claim 1, wherein the step of initiating an audit process is performed as soon as a threshold number of results are stored into the queue.

7. The method of claim 6, further comprising the step of pausing the audit process when the number of results stored in the queue is less than a minimum number.

8. A system for assessing the vulnerability of a target object, the system comprising:

a memory device;

a crawler process, communicatively coupled to the memory device and operative to conduct a crawl of the target object and store the results of the crawl into the memory device;

a scanner process, communicatively coupled to the crawler process and operative to invoke the crawler process by providing the identity of the target object;

an audit process, communicatively coupled to the memory device and operative to conduct an audit simultaneously with the operation of the crawler process by extracting the results of the crawler process from the memory device while the crawler process continues to add new results to the memory device; and

a plurality of audit engines invoked by the audit process and operative to perform various attacks on the target object,

wherein the audit process is further operative to obtain attack results from the plurality of audit engines and provide at least a portion of the attack results as further input to the crawler process while the crawler process is still operating, and

wherein the audit process is further operative to store the attack results into the memory device and the scanner process is further operative to allocate processing power between the crawling process and the auditing process based at least in part on the number of crawling results and attack results stored in the memory device.

9. The system of claim 8, wherein the at least a portion of the attack results provided to the crawler process are those attack results that reveal structure regarding the target object.

10. The system of claim 8, further comprising a user interface, the scanner is communicatively coupled to the user interface and the audit process and is further operative to receive attack results from the crawler process and the audit process and provide these results to the user interface, whereby the results of the audit process are available before the crawling process is completed.

11. The system of claim 8, wherein the scanner process is further operative to allocate processing power between the crawling process and the auditing process.

12. The system of claim 8, wherein the audit process is operative to conduct the audit simultaneously with the operation of the crawler process only after the crawling process has stored a threshold number of crawling results into the memory device.

13. The system of claim 12, wherein the audit process is further operative to pause when the number of crawling results stored in the memory device is less than a minimum number.

14. A method for performing a vulnerability assessment of a target object and providing results thereof to a user, the method comprising the steps of:

receiving input identifying a target object to assess;

conducting a crawling process on the identified object;

storing the results of the crawling process into a queue in a memory storage device;

conducting an audit process while the crawling process is still in operation and results are still being stored into the memory storage device;

extracting one or more of the stored results of the crawling process from the memory storage device and providing them to the audit process;

storing the results of the audit process into the memory storage device;

extracting the stored results of the audit process from the memory storage device and providing them to the crawling process, while the crawling process is still in operation, for further crawling; and

allocating processing power to the co-operating crawling process and the audit process based on the number of results existing in the queue.

15. The method of claim 14, wherein the step of conducting an audit process is initiated once a threshold number of results from the crawling process have been stored in the memory storage device.