Method and System for Log File Analysis Based on Distributed Computing Network

• US 20100088354A1
• Filed: 09/29/2007
• Published: 04/08/2010
• Est. Priority Date: 11/30/2006
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Li, Jianghua, Wu, Weixun, Li, Ang, Zhang, Jinyin
• Granted Patent: US 8,671,097 B2
• Time in Patent Office: Days
• US Class Current: 707/827
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

The present invention discloses a method and a system for log file analysis based on distributed computing network. The method includes: storing user identifiers and related log information into a log file; dividing the log file into target files each including the log information having the same user identifier; separately analyzing the target files to obtain analysis results using at least two nodes; and combining the analysis results of the nodes. The method thereby establishes relationships among various log files through user identifiers, and further analyzes the relationships among the user'"'"'s accesses to various contents of a website.

Claims

1. A method for log file analysis based on a distributed computing network, characterized in that the method comprises:

storing user identifiers and related log information into a log file;

dividing the log file into a plurality of target files, wherein log information having a common user identifier is included in the same target file;

separately analyzing the plurality of target files to obtain analysis results using two or more nodes; and

combining the analysis results of the nodes.

2. The method for log file analysis as recited in claim 1, characterized in that the log file is a log file of a web server.

3. The method for log file analysis as recited in claim 2, characterized in that the web server is a server cluster formed by at least one server, and the log file is a log file set including log files created by the servers in the server cluster.

4. The method for log file analysis as recited in claim 1 or 3, characterized in that dividing the log file into a plurality of target files comprises:

downloading the log file into a log analysis server;

sending the log file to the nodes by the log analysis server;

dividing by each node the received log file into identifier files according to the user identifiers, placing the log information having the same user identifier into the same identifier file, and sending the identifier files to the log analysis server; and

collecting by the log analysis server the identifier files sent from the nodes and combining identifier files having the same user identifier into a single file to form the corresponding target file.

5. The method for log file analysis as recited in claim 4, characterized in that prior to dividing the log file into identifier files in each node, the method further comprises:

filtering out information that is unrelated to log analysis from the log file.

6. The method for log file analysis as recited in claim 5, characterized in that prior to analyzing the target files in the nodes, the method further comprises:

ordering the log information in the log file according to times of creation of the log information.

7. The method for log file analysis as recited in claim 1, characterized in that each user identifiers is created by the web server according to the associated user'"'"'s IP and a random number, and is stored in the user'"'"'s local machine.

8. The method for log file analysis as recited in claim 7, characterized in that the user identifier is stored in a cookie of the user'"'"'s local machine.

9. A system for log file analysis based on a distributed computing network, characterized in that the system comprises a log analysis server and a plurality of nodes, wherein the log analysis server comprises:

a collection unit used for collecting a log file of a web server, the log file including user identifiers and log information related to the user identifiers;

a storage unit used for storing the log file collected by the collection unit;

a first interface unit used for receiving and sending data; and

a division unit used for dividing the log file in the storage unit into multiple target files, wherein the log information having a common user identifier is included in the same target file;

the nodes comprise;

a second interface unit used for receiving and sending data; and

a processing unit used for analyzing the target files;

and the log analysis server further comprises;

a combination unit used for combining analysis results of the plurality of nodes.

10. The system for log file analysis as recited in claim 9, characterized in that the log file is of a web server, which is a server cluster comprising at least one server, and the log file is a log file set including log files created by each server in the server cluster.

11. The system for log file analysis as recited in claim 10, wherein each node further comprises:

a filtering unit used for filtering out information that is unrelated to log analysis from the log file; and

a partitioning unit used for dividing the filtered log file into identifier files according to the user identifiers and placing the log information having the same user identifier into the same identifier file;

and the division unit further comprises;

a target file generation unit used for combining the identifier files sent from the nodes which have the same user identifier into a single file to form the respective target file.

12. The system for log file analysis as recited in claim 11, characterized in that nodes further comprise:

an ordering unit used for ordering the log information in the log file according to times of creation of the log information.