System for collecting, analyzing, and reporting high volume multi-web server usage

• US 6,789,115 B1
• Filed: 07/09/1999
• Issued: 09/07/2004
• Est. Priority Date: 07/09/1999
• Status: Expired due to Term

• Current Assignee: Bank of America Corp.
• Original Assignee: Merrill Lynch & Co. Incorporated
• Inventors: Rico-Connelly, Susan, Shmuelov, Akiva, Nayak, Vinith, Singer, Barry, Gable, Glen, Sherman, J. David
• Primary Examiner(s): Etienne, Ario
• Assistant Examiner(s): SALAD, ABDULLAHI ELMI
• Application Number:
  US09/346,145
• Time in Patent Office: 1,887 Days
• Field of Search: 709/224, 709/223, 709/217, 709/219, 709/226, 707/102, 707/201, 707/10
• US Class Current: 709/224
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A system for capturing, analyzing, storing and reporting system-users'"'"' usage of multiple internet and/or intranet web servers. At each web server in the system, the actions by the system-user create a server log that is processed by the system on a continuous basis. The processing at the server may include, by way of illustration, filtering the logs of about 75 percent of the data and decrypting a system-user'"'"'s cookie. On a periodic basis, one or more collection servers in the system copy the processed log files from each web server on the system, zip the files, and transfer the files to an analysis server. After the collection is complete, an analysis server processes the data and stores it in a relational database that supports various user specified daily, monthly and quarterly reports of the usage data.

Claims

1. A system for collecting, processing, and reporting users content requests at one or more web servers, comprising:

a log generating component for generating server logs and capturing content requests made by users of one or more web servers;

a filtering component operative to perform real-time filtering out of specified data types from said server logs so as to provide processed server logs;

an analysis server containing a database comprising;

a main table operative to store a unique identifier for each users content request, and

one or more reference tables operative to store each of said unique identifiers and to associate each identifier with a corresponding content request;

a collection component operative to;

retrieve said processed server logs and content requests,

locate in said one or more reference tables unique identifiers corresponding to said retrieved content requests, and

load said located unique identifiers into said main table; and

a display component operative to display information stored in said database.

2. A system as in claim 1, wherein said specified data types consist of graphic image files.

3. A system as in claim 1, wherein said specified data types comprise graphic image files.

4. A system as in claim 1, wherein said database is operative to decompress incoming compressed files and to test and discard unrecognized formats.

5. A system as in claim 1, wherein said database is operative to reformat log data for processing by SQL server bulk copy load.

6. A system as in claim 5, wherein said reformatting comprises removing unused fields, changing time stamps, and applying field separators.

7. A system as in claim 1, wherein said database is operative to use a bulk copy load utility to place data into a temporary SQL table.

8. A system as in claim 1, wherein said database is operative to run an import program that tests for a unique identifier.

9. A system as in claim 1, wherein said database further comprises a summary table configured to contain aggregated information regarding functions and usage, wherein said aggregated information is collected over selected time intervals.

10. A system as in claim 1, wherein said main table is configured to contain composite data corresponding to a selected time interval, wherein said composite data is mapped to unique identifiers to facilitate aggregation reports.

11. A system as in claim 1, wherein said one or more reference tables store actual data linked, via unique identifiers, to said main table.

12. A system as in claim 1, further comprising a local data collection server that is in communication with said one or more web servers and also in communication with said analysis server.

13. A system as in claim 12, wherein said analysis server is remotely located from said local data collection server and said one or more web servers.

14. A system as in claim 13, wherein said filtering component is located at said local data collection server and wherein said processed server logs are collected at selected time intervals by said local data collection server and compressed into compressed files for transmission to said remotely located analysis server.