INTEGRATED CRAWLING AND AUDITING OF WEB APPLICATIONS AND WEB CONTENT

• US 20070061877A1
• Filed: 08/01/2006
• Published: 03/15/2007
• Est. Priority Date: 02/11/2004
• Status: Active Grant

• Current Assignee: Micro Focus LLC (Open Text Corporation)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Sullivan, Bryan, Millar, Steve, Kelly, Raymond, Tillery, David, Sima, Caleb, Sullivan, Gerald
• Granted Patent: US 7,765,597 B2
• Time in Patent Office: Days
• US Class Current: 726/12
• CPC Class Codes: H04L 63/12 Applying verification of the received information cryptographic mechanisms or cryptographic arrangements for data integrity or data verification H04L9/32
• CPC Class Codes: H04L 63/20 for managing network security; network security policies in general filtering policies H04L63/0227

Abstract

A vulnerability assessment tool that is operative to analyze web sites by simultaneously operating a crawling process and an audit process. Once the crawling process is invoked, the results are provided to the audit process. The audit process, rather than waiting until the crawl process is completed, simultaneously audits the web site based on the already provided crawl results. The results of the audit are also fed back to the crawl process to further enhance the crawl.

Claims

1. A method for performing a vulnerability assessment of a target object, the method comprising the steps of:

identifying a target object to assess;

initiating a crawling process on the identified object;

storing the results of the crawling process into a memory storage device;

initiating an audit process while the crawling process is still in operation;

extracting the stored results of the crawling process from the memory storage device and providing them to the audit process.

2. The method of claim 1, further comprising the steps of:

storing the results of the audit process into the memory storage device; and

extracting the stored results of the audit process from the memory storage device and providing them to the crawling process for further crawling.

3. The method of claim 1, wherein the target object is a web site and the step of identifying the target object comprises identifying a URL for the web site and the step of initiating a crawling process further comprises providing the URL to the crawling process.

4. The method of claim 3, wherein the step of initiating a crawling process further comprises examining the HTML and objects associated with the URL.

5. The method of claim 1, wherein the target object is a web application accessible at a URL, and the step of initiating a crawling process involves examining the web application.

6. The method of claim 1, wherein the step of initiating a crawling process comprises searching the target object for links, sessions and files.

7. The method of claim 1, wherein the step of storing the results of the crawling process into a memory storage device further comprises storing the results into a queue.

8. The method of claim 7, wherein the step of initiating an audit process is performed as soon as a threshold number of results are stored into the queue.

9. The method of claim 8, further comprising the step of pausing the audit process when the number of results stored in the queue is less than a minimum number.

10. The method of claim 1, wherein the step of storing the results of the crawling process into a memory storage device further comprises storing the results into a queue and further comprising the step of allocating processing power to the crawling process and the audit process based on the number of results existing in the queue.

11. A system for assessing the vulnerability of a target object, the system comprising:

a memory device;

a crawler process, communicatively coupled to the memory device and operative to conduct a crawl of the target object and store the results of the crawl into the memory device;

a scanner process, communicatively coupled to the crawler process and being operative to invoke the crawler process by providing the identity of the target object;

an audit process, communicatively coupled to the memory device and operative to conduct an audit simultaneously with the operation of the crawler process by extracting the results of the crawl from the memory device; and

a plurality of audit engines invoked by the audit process and operative to perform various attacks on the target object.

12. The system of claim 11, wherein the audit process is further operative to obtain attack results from the plurality of audit engines and provide at least a portion of the attack results as further input to the crawler process.

13. The system of claim 12, wherein the at least a portion of the attack results provide to the crawler process are those attack results that reveal structure regarding the target object.

14. The system of claim 11, further comprising a user interface, the scanner is communicatively coupled to the user interface and the audit process and is further operative to receive attack results from the crawler process and the audit process and provide these results to the user interface, whereby the results of the audit process are available before the crawling process is completed.

15. The system of claim 11, wherein the scanner process is further operative to allocate processing power between the crawling process and the auditing process.

16. The system of claim 11, wherein the audit process is further operative to store the attack results into the memory device and the scanner process is further operative to allocate processing power between the crawling process and the auditing process based at least in part on the number of crawling results and attack results stored in the memory device.

17. The system of claim 11, wherein the audit process is operative to conduct the audit simultaneously with the operation of the crawler process only after the crawling process has stored a threshold number of crawling results into the memory device.

18. The system of claim 17, wherein the audit process is further operative to pause when the number of crawling results stored in the memory device is less than a minimum number.

19. A method for performing a vulnerability assessment of a target object and providing results thereof to a user, the method comprising the steps of:

receiving input identifying a target object to assess;

conducting a crawling process on the identified object;

storing the results of the crawling process into a memory storage device;

conducting an audit process while the crawling process is still in operation;

extracting the stored results of the crawling process from the memory storage device and providing them to the audit process.

storing the results of the audit process into the memory storage device; and

extracting the stored results of the audit process from the memory storage device and providing them to the crawling process for further crawling.

20. The method of claim 19, wherein step of conducting an audit process is initiated once a threshold number of results from the crawling process have been stored in the memory storage device.