Session-based processing method and system

• US 7,779,021 B1
• Filed: 03/09/2004
• Issued: 08/17/2010
• Est. Priority Date: 03/09/2004
• Status: Active Grant

• Current Assignee: Versata Development Group, Inc. (ESW Capital, LLC)
• Original Assignee: Versata Development Group, Inc. (ESW Capital, LLC)
• Inventors: Smith, Shawn A. P., Karipides, Daniel P.
• Primary Examiner(s): Wong; Don
• Assistant Examiner(s): Hwa; Shyue Jiunn
• Application Number:
  US10/796,317
• Time in Patent Office: 2,352 Days
• Field of Search: None
• US Class Current: 707/760
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A log file processing system sorts records from large log files and groups them by session without making a complete copy of the log files by capturing a subset of the log files in a sliding memory window and identifying all records in the window that form a complete user session. Records belonging to a complete session are output for analyzing, and the remaining records are output as raw log data for additional processing. Using a ring buffer to implement the sliding memory window, data structures are used to group records by session, to identify completed sessions, and to index into the ring buffer to retrieve records for completed sessions that are to be directly analyzed. Any records remaining in the ring buffer at the end of slide window processing may be output as raw log file data and are processed as incomplete or malformed session records. An embodiment of the log file processing system provides a significant improvement on the speed of data extraction from log files into analyzable session data.

Claims

1. A method for grouping log file entries by session, comprising:

storing a log file of entries in a memory, each of said entries identifying a client request to a server;

retrieving a subset of log file entries from the memory for storage in a ring buffer;

processing each entry in the ring buffer to identify entries in the subset of log file entries that belong to a complete client session by allocating, for each identified client session, an index to identify entries in the ring buffer that are associated with the identified client session and to identify start or end entries;

grouping entries in the subset that belong to a complete client session; and

adding and removing log file entries to the ring buffer so that the ring buffer implements a sliding window to process the log file entries in the memory into complete client sessions.

2. The method of claim 1, wherein a complete client session is identified by identifying all entries in the subset that are associated with a particular client session and that include both a beginning entry and an end entry.

3. The method of claim 2, wherein an end entry is identified as any entry that corresponds to a logout request.

4. The method of claim 2, wherein an end entry for a client session is identified as any entry associated with that client session that has no other entries for that client session that occur within a session expiration window.

5. The method of claim 2, wherein an end entry for a client session is identified as any entry having a first timestamp value, where the difference between first timestamp value and a second timestamp value associated with a subsequent entry in the subset of log files exceeds a timeout value.

6. The method of claim 1, further comprising outputting all entries in the subset of log file entries that do not belong to a complete client session as raw log data.

7. The method of claim 1, further comprising outputting as raw log data all entries in the subset of log file entries that belong to an incomplete client session which has a beginning entry but no end entry.

8. An article of manufacture having at least one recordable medium having stored thereon executable instructions and data which, when executed by at least one processing device, cause the at least one processing device to:

read a plurality of records from a file system into a ring buffer, where said plurality of records comprises a subset of all records in the file system;

scan each record in the ring buffer to identify a user session for said record and to identify any start or end records in the ring buffer;

allocate, for each identified user session, an index to identify all records in the ring buffer that are associated with the identified user session and to identify all start or end records;

process the index to group all records in the ring buffer belonging to a complete user session, to output the grouped records for further analysis; and

sequentially adding and removing log records to the ring buffer until all of the log records in the file system have been processed so that the ring buffer implements a sliding window to process all of the log records in the file system into complete user sessions.

9. The article of manufacture of claim 8, wherein the index comprises:

a session record for each identified user session for keying into the ring buffer to identify log records associated with said identified user session;

a hash table for keying into the session record based upon session key information;

a linked listing of last seen log records for each session; and

a linked list of first seen log records for each session.

10. A system for session-based processing of log files using a data processing system and network session data collected from one or more users, the system comprising:

a log file collection system for collecting a plurality of server request entries, wherein a server request entry comprises a session identifier; and

a processing engine to process a subset of the plurality of server request entries to group the server request entries by session using the session identifier in each server request entry by;

reading the subset of the plurality of server request entries from the log file collection into a ring buffer;

scanning each server request entry in the ring buffer to identify a user session for said server request entry and to identify any start or end entries in the ring buffer;

allocating, for each identified user session, an index to identify server request entries in the ring buffer that are associated with the identified user session and to identify start or end entries;

processing the index to group server request entries in the ring buffer belonging to a complete user session; and

adding and removing server request entries to the ring buffer so that the ring buffer implements a sliding window to process the server request entries in the log file collection system into complete user sessions.

11. The system of claim 10, wherein the processing engine uses a plurality of data structures to group the web server request entries by session, said plurality of data structures comprising:

a ring buffer for storing the subset of the plurality of web server request entries,

a per-session record for keying into the ring buffer

a hash table for keying into the per-session records

a linked list of last processed web server request entries for each session, and

a linked list of first processed web server request entries for each session.

12. The system of claim 10, wherein the processing engine uses a sliding memory window to process the subset of the plurality of web server request entries.

13. The system of claim 10, further comprising a parser for further analysis the web server request entries that have been grouped by session to generate a user session history.

14. The system of claim 10, where the processing engine generates an output file containing web server request entries corresponding to one or more complete user sessions.

15. The system of claim 10, where the processing engine generates an output file containing web server request entries corresponding to one or more incomplete user sessions.

16. The system of claim 10, where the processing engine generates an output file containing web server request entries corresponding to one or more user sessions that do not include an end session entry.

17. A system for parsing web site logs one session at a time, comprising:

means for storing network session data from at least one server log file;

means for reading a subset of the network session data into a ring buffer;

means for processing the subset of the network session data in the ring buffer to group said network session data by session by allocating for each identified user session, an index to identify network session data in the ring buffer that is associated with the identified user session and to identify start or end network session data; and

means for generating a first output file containing network session data grouped by session by processing the index to group network session data in the ring buffer belonging to a complete user session;

means for adding and removing network session data to the ring buffer so that the ring buffer implements a sliding window to process the network session data into complete user sessions; and

means for parsing said first output file.

18. The system of claim 17, wherein the means for reading a subset of the network session data comprises a sliding window.

19. The system of claim 17, wherein the means for reading a subset of the network session data comprises a ring buffer.