Method and system for log file analysis based on distributed computing network

• US 8,671,097 B2
• Filed: 09/29/2007
• Issued: 03/11/2014
• Est. Priority Date: 11/30/2006
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holdings Limited
• Inventors: Wu, Weixun, Li, Jianghua, Zhang, Jinyin, Li, Ang
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US12/516,868
• Publication Number:
  US 20100088354A1
• Time in Patent Office: 2,355 Days
• Field of Search: 707/827, 707/672, 707/722, 707/736, 707/999.007
• US Class Current: 707/736
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

The present invention discloses a method and a system for log file analysis based on distributed computing network. The method includes: storing user identifiers and related log information into a log file; dividing the log file into target files each including the log information having the same user identifier; separately analyzing the target files to obtain analysis results using at least two nodes; and combining the analysis results of the nodes. The method thereby establishes relationships among various log files through user identifiers, and further analyzes the relationships among the user'"'"'s accesses to various contents of a website.

Claims

1. A method for log file analysis based on a distributed computing network system, characterized in that the method comprises:

storing, in a log file, a plurality of user identifiers and log information associated with the plurality of user identifiers;

dividing the log file into a plurality of target files such that each of the plurality of target files includes log information associated with a user identifier of the plurality of user identifiers;

prior to dividing the log file into identifier files, filtering out information that is unrelated to log analysis from the log file and ordering it according to the time of the creation of the log information;

separately analyzing the plurality of target files to obtain analysis results using a plurality of nodes, the distributed computer network system including the plurality of nodes and a log analysis server; and

combining analysis results of the plurality of nodes,

wherein dividing the log file into a plurality of target files comprises;

downloading the log file into the log analysis server of the distributed computing network;

sending the log file to the plurality of nodes by the log analysis server;

dividing by each node of the plurality of nodes the log file into identifier files according to the plurality of user identifiers;

placing the log information having a same user identifier into a same identifier file;

sending the identifier files to the log analysis server;

collecting by the log analysis server the identifier files sent from the plurality of nodes; and

combining identifier files having the same user identifier into a single file to form a corresponding target file.

2. The method for log file analysis as recited in claim 1, wherein the log file is a log file of a web server.

3. The method for log file analysis as recited in claim 2, wherein the web server is a server cluster formed by at least one server, and the log file is a log file set including log files created by the servers in the server cluster.

4. The method for log file analysis as recited in claim 1, further comprising:

ordering the log information in the log file according to times of creation of the log information.

5. The method for log file analysis as recited in claim 1, wherein each user identifier is created by the web server according to the associated user'"'"'s IP and a random number, and is stored in the user'"'"'s local machine.

6. The method for log file analysis as recited in claim 5, wherein the user identifier is stored in a cookie of the user'"'"'s local machine.

7. A system comprising:

one or more processors;

and memory storing a log analysis unit executable by the one or more processors to;

dividing a log file into a plurality of target files such that each of the plurality of target files includes log information associated with a user identifier of a plurality of user identifiers;

prior to dividing the log file into identifier files, filtering out information that is unrelated to log analysis from the log file and ordering it according to the time of the creation of the log information;

transmit multiple log files to a plurality of nodes of a computing network based on content of each log file of the multiple log files, the multiple log files including log information associated with the plurality of user identifiers,

combine divided log files received from the plurality of nodes to obtain multiple target files each associated with a user identifier of the plurality of user identifiers,

send the multiple target files to the plurality of nodes, and

combine analysis results received from the plurality of nodes to obtain an analysis result for the multiple log files; and

the plurality of nodes configured to;

divide each of the multiple log files received from the log analysis unit to obtain the divided log files based on the plurality of user identifiers,

placing the log information having a same user identifier into a same identifier file;

send the divided log file to the log analysis unit,

analyze the multiple target files to obtain the analysis results, and

send the analysis results to the log analysis unit.

8. The system as recited in claim 7, wherein the transmitting of the multiple log files to the plurality of nodes of the server cluster comprises transmitting the multiple log files based on content of each of the multiple log files, the content being related to at least one of news, forums and blogs.

9. The system as recited in claim 7, wherein the sending of the multiple target files to the plurality of nodes comprises sending each of the multiple target files to a node of the plurality of nodes.

10. The system as recited in claim 7, wherein the analyzing the multiple target files to obtain the analysis results comprises analyzing, by a node of the plurality of nodes, each of the multiple log files to obtain one analysis result.

11. The system as recited in claim 7, wherein the plurality of nodes are further configured to sort the log information associated with the multiple target files based on a time that the log information associated with the multiple target files is created.

12. The system as recited in claim 7, wherein the plurality of nodes are further configured to remove, from the multiple log files, information that is not related to log analysis, the information being determined based on an Internet Protocol (IP) address associated with the information in a corresponding log file.

13. The system as recited in claim 7, wherein the plurality of nodes are further configured to sort the log information of a target file of the multiple target files based on a historical sequence of user activities of a user while the user visits a website, a user identifier of the user being associated with the target file.

14. The system as recited in claim 13, wherein the user activities include a click-through on an advertisement associated with the website or a purchase of one or more items associated with the advertisement.

15. The system as recited in claim 7, wherein the plurality of nodes are further configured to sort the log information of a target file of the multiple target files based on an analysis of an efficiency of an advertisement associated with a website that a user visits, a user identifier of the user being associated with the target file.