Session-based processing method and system
First Claim
1. A method for grouping log file entries by session, comprising:
- storing a log file of entries in a memory, each of said entries identifying a client request to a server;
retrieving a subset of entries from the memory for storage in a ring buffer that is smaller than the memory;
processing each entry in the memory to identify entries that belong to a complete client session by using the ring buffer to implement a sliding window to process all entries in the log file into complete client sessions by sequentially adding and removing entries to the ring buffer until all of the entries in the log file have been processed; and
outputting all entries in the subset of log file entries that do not belong to the complete client session as raw log data.
0 Assignments
0 Petitions
Accused Products
Abstract
A log file processing system sorts records from large log files and groups them by session without making a complete copy of the log files by capturing a subset of the log files in a sliding memory window and identifying all records in the window that form a complete user session. Records belonging to a complete session are output for analyzing, and the remaining records are output as raw log data for additional processing. Using a ring buffer to implement the sliding memory window, data structures are used to group records by session, to identify completed sessions, and to index into the ring buffer to retrieve records for completed sessions that are to be directly analyzed. Any records remaining in the ring buffer at the end of slide window processing may be output as raw log file data and are processed as incomplete or malformed session records. An embodiment of the log file processing system provides a significant improvement on the speed of data extraction from log files into analyzable session data.
-
Citations
17 Claims
-
1. A method for grouping log file entries by session, comprising:
-
storing a log file of entries in a memory, each of said entries identifying a client request to a server; retrieving a subset of entries from the memory for storage in a ring buffer that is smaller than the memory; processing each entry in the memory to identify entries that belong to a complete client session by using the ring buffer to implement a sliding window to process all entries in the log file into complete client sessions by sequentially adding and removing entries to the ring buffer until all of the entries in the log file have been processed; and outputting all entries in the subset of log file entries that do not belong to the complete client session as raw log data. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. An article of manufacture comprising a non-transitory computer readable medium having stored thereon executable instructions and data which, when executed by at least one processing device, cause the at least one processing device to:
-
read a plurality of records from a file system into a ring buffer, where said plurality or records comprises a subset of all records in the file system; scan each record in the ring buffer to identify a user session for said record and to identify any start or end records in the ring buffer; allocate, for each identified user session, an index to identify all records in the ring buffer that are associated with the identified user session and to identify all start or end records, where the index comprises; a session record for each identified user session for keying into the ring buffer to identify log records associated with said identified user session, a hash table for keying into the session record based upon session key information, a linked listing of last seen log records for each session, and a linked list of first seen log records for each session; and process the index to group all records in the ring buffer belonging to a complete user session, to output the grouped records for further analysis. - View Dependent Claims (8)
-
-
9. A system for session-based processing of log files using a data processing system and network session data collected from one or more users, the system comprising:
-
a log file collection system for storing a plurality of server request entries in memory, wherein a server request entry comprises a session identifier; a processing engine to process a subset of the plurality of server request entries to group the server request entries by session using the session identifier in each server request entry by reading the plurality of server request entries into a ring buffer that is smaller than the memory to implement a sliding window to process the plurality of server request entries into complete sessions by sequentially adding and removing server request entries to the ring buffer until all of the plurality of server request entries in the log file collection system have been processed; and a parser for analysis of the web server request entries that have been grouped by session to generate a user session history. - View Dependent Claims (10, 11, 12, 13, 14)
-
-
15. A system for parsing web site logs one session at a time, comprising:
-
means for storing network session data from at least one server log file; means for processing a subset of the network session data to group said network session data by session using a ring buffer to implement a sliding window which loads a programmably adjustable portion of the network session data from at least one server log file to process network session data into one or more complete sessions by sequentially adding and removing server request entries from the network session data to the ring buffer until all of the network session data from the at least one server log file has been processed; means for generating a first output file containing network session data grouped by session; and means for parsing said first output file. - View Dependent Claims (16, 17)
-
Specification