System and method for monitoring and analyzing internet traffic

• US 6,792,458 B1
• Filed: 10/04/2000
• Issued: 09/14/2004
• Est. Priority Date: 10/04/1999
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Urchin Software Corporation (Alphabet Inc.)
• Inventors: Muret, Paul Nicolas, Moon, Hui Sok
• Primary Examiner(s): Etienne, Ario
• Assistant Examiner(s): JACOBS, LASHONDA T
• Application Number:
  US09/679,297
• Time in Patent Office: 1,441 Days
• Field of Search: 709/224, 709/223, 709/225, 709/226, 709/227, 709/228, 709/229, 709/217-19
• US Class Current: 709/224
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/06 Generation of reports
• CPC Class Codes: H04L 43/062 related to network traffic
• CPC Class Codes: H04L 43/065 related to network devices
• CPC Class Codes: H04L 43/067 using time frame reporting
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps

Abstract

A system and method for monitoring and analyzing Internet traffic is provided that is efficient, completely automated, and fast enough to handle the busiest websites on the Internet, processing data many times faster than existing systems. The system and method of the present invention processes data by reading log files produced by web servers, or by interfacing with the web server in real time, processing the data as it occurs. The system and method of the present invention can be applied to one website or thousands of websites, whether they reside on one server or multiple servers. The multi-site and sub-reporting capabilities of the system and method of the present invention makes it applicable to servers containing thousands of websites and entire on-line communities. In one embodiment, the system and method of the present invention includes e-commerce analysis and reporting functionality, in which data from standard traffic logs is received and merged with data from e-commerce systems. The system and method of the present invention can produce reports showing detailed “return on investment” information, including identifying which banner ads, referrals, domains, etc. are producing specific dollars.

Claims

1. A system for analyzing and monitoring internet traffic, comprising:

a relational database; and

a log engine that processes log files received from at least one internet server and stores data processed from the log files in the relational database;

wherein the log engine, when new log file data is present in the log file, processes said new log file data and determines an “

end of file”

location on the log file, and, when new log file data is not present in the log file, periodically checks the log file at predetermined time intervals to check for new log file data, and commences processing of any new log file data at a most recent determined “

end of file”

location on the log file.

2. The system of claim 1, wherein the relational database comprises a plurality of hash tables.

3. The system of claim 1, wherein the plurality of tables comprise:

a visitor table that stores traffic information generated by a visitor to an internet site hosted by the at least one internet server; and

a plurality of data tables, wherein each data table stores records related to a respective parameter.

4. The system of claim 3, wherein the visitor table comprises at least one pointer to at least one record stored in at least one of the data tables.

5. The system of claim 3, wherein the respective parameters comprise:

domain names from which the visitor originated; and

web browsers used by the visitor; and

other internet sites that referred the visitor to the internet site.

6. A system for analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, comprising:

a visitor centric database; and

a log engine that receives log files from the at least one internet server, processes hits logged in each log file, and stores traffic data derived from the hits in the visitor centric database, wherein the visitor centric database associates the traffic data derived from the hits with a visitor that generated the hit;

wherein the log engine, when new log file data is present in the log file, processes said new log file data and determines an “

end of file”

location on the log file, and, when new log file data is not present in the log file, periodically checks the log file at predetermined time intervals to check for new log file data, and commences processing of any new log file data at a most recent determined “

end of file”

location.

7. The system of claim 6, wherein the visitor centric database comprises a plurality of hash tables.

8. The system of claim 6, wherein the plurality of hash tables comprise:

a visitor table that stores traffic information derived from the hits, wherein the visitor table contains a unique visitor record for each visitor; and

a plurality of data tables, wherein each data table stores data related to a respective non-unique parameter.

9. The system of claim 8, wherein the visitor table comprises at least one pointer to at least one record stored in at least one of the data tables.

10. The system of claim 8, wherein the respective non-unique parameters comprise:

domain names from which the visitors originated;

web browsers used by the visitors; and

other internet sites that referred the visitors to the at least one internet site.

11. The system of claim 8, wherein the log engine comprises a visitor identifier that determines if a hit originates from a new visitor or an existing visitor.

12. The system of claim 11, wherein the visitor identifier is adapted to create a new visitor record if a hit originates from a new visitor.

13. The system of claim 6, wherein the log engine comprises a database buffer that temporarily stores the traffic data derived from the hits logged in the log files.

14. The system of claim 11, wherein the log engine further comprises a database updater that transfers the traffic data temporarily stored in the database buffer to the visitor centric database.

15. The system of claim 12, wherein the database updater sorts the traffic data temporarily stored in the database buffer before transferring the traffic data to the visitor centric database.

16. The system of claim 6, wherein the log engine comprises a log parser that reads log lines in the log files, and separates each log file into individual fields.

17. The system of claim 6, further comprising a report engine that generates reports using the traffic data stored in the visitor centric database.

18. The system of claim 17, wherein the report engine is adapted to generate reports that correlate money spent by a visitor to any other parameter of the traffic data.

19. The system of claim 17, wherein the report engine is adapted to generate a top products report that ranks products purchased by visitors based on revenues generated by the products.

20. The system of claim 17, wherein the report engine is adapted to generate at least one of a totals report, product tree report, regions report, and top scores report.

21. The system of claim 17, wherein the report engine is adapted to generate a report that displays a value of at least one traffic data parameter over at least one predetermined time period.

22. The system of claim 21, wherein the report comprises a snapshot report in which the at least one predetermined time period comprises seven consecutive 24 hour time periods.

23. The system of claim 21, wherein the report comprises an hourly graph report in which the at least one predetermined time period comprises a plurality of consecutive one hour time periods.

24. The system of claim 17, wherein the report engine is adapted to generate a top pages report that ranks website pages based on number of visitors to the website pages.

25. The system of claim 24, wherein each entry in the top pages report comprises a link for accessing additional information about a respective website page.

26. The system of claim 17, wherein the report engine is adapted to generate a search engine report that displays a list of most used search engines.

27. The system of claim 17, wherein the report engine is adapted to generate a top domains report that displays regional and network information about the visitors.

28. The system of claim 17, wherein the report engine is adapted to generate a browser tree report that ranks internet browsers based on which internet browsers are used most by visitors to a website.

29. The system of claim 28, wherein each internet browser entry in the browser tree report includes a link for accessing information about different versions of a respective internet browser.

30. The system of claim 17, wherein the report engine is adapted to generate a top entrances report that ranks starting points of visitors to a website based most used starting points.

31. The system of claim 17, wherein the report engine is adapted to generate at least one of a summary report, a daily graph report, a monthly graph report, a top servers report, a file types report, a status/errors report, a posted forms report, a top referrals report, a top keywords report, a referral tree report, a domain tree report, a top countries report, a platform tree report, a top combos report, a top exits report, a click through report, a depth of visit report, a length of visit report, and a usernames report.

32. The system of claim 17, wherein the report engine comprises:

a template module that stores report templates;

a session parser that receives report requests from the at least one server, and determines a type of report requested, data needed to generate a requested report and a format for the requested report;

an authenticator that receives an identity of a report requester from the session parser, and verifies that the report requester has permission to view a requested report;

a data query module that receives authentication information from the authenticator, and that queries the database for data needed to generate the requested report if the report requester has permission to view the requested report; and

a format output module that receives the data needed to generate the requested report from the database, retrieves templates for the requested report from the template module, creates the requested report, and delivers the requested report to the report requester.

33. The system of claim 32, wherein the template module also stores at least one dictionary.

34. The system of claim 33, wherein the format output module is adapted to create the requested report in a selectable language using the at least one dictionary.

35. The system of claim 6, wherein the log engine is configured to process hits from multiple internet sites that are logged to a single log file.

36. The system of claim 6, wherein the log engine comprises a website identifier that identifies a source of each hit.

37. The system of claim 6, wherein the log engine comprises a domain name system (DNS) resolver that determines host and domain information for each visitor.

38. The system of claim 37, wherein the DNS resolver utilizes reverse DNS resolution to determine the host and domain information for each visitor.

39. The system of claim 6, wherein the log engine is adapted to process e-commerce log files that contain information on money spent by a visitor.

40. An article of manufacture, comprising:

a computer usable medium having computer readable program code embodied therein for analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, the computer readable program code in the article of manufacture comprising;

computer readable program code for receiving log files from the at least one internet server;

computer readable program code for processing hits logged in each log file by;

initiating a process loop when new data is present in the log file, during which the new data is processed and an “

end of file”

location is determined for use as a starting point for subsequent new data processing, and

initiating a wait loop when new data is not present in the log file, wherein the wait loop delays data processing for a predetermined time interval before checking for new data in the log file;

computer readable program code for storing traffic data derived from the hits in a database; and

computer readable program code for associating the traffic data derived from the hits and stored in the database with a visitor that generated the hit.

41. The article of manufacture of claim 40, wherein the database comprises a plurality of hash tables.

42. The article of manufacture of claim 41, wherein the plurality of hash tables comprise:

a visitor table that stores traffic information derived from the hits, wherein the visitor table contains a unique visitor record for each visitor; and

a plurality of data tables, wherein each data table stores data related to a respective non-unique parameter.

43. The article of manufacture of claim 42, wherein the visitor table comprises at least one pointer to at least one record stored in at least one of the data tables.

44. The article of manufacture of claim 42, wherein the computer readable program code for processing hits logged in each log file comprises computer readable program code for determining if a hit originates from a visitor with a preexisting visitor record in the database.

45. The article of manufacture of claim 44, wherein the computer readable program code for determining if a hit originates from a visitor with a preexisting visitor record in the database creates a new visitor record if a hit originates from a visitor without a preexisting visitor record in the database.

46. The article of manufacture of claim 40, further comprising computer readable program code for temporarily storing the traffic data derived from the hits logged in the log files.

47. The article of manufacture of claim 46, wherein the computer readable program code for storing traffic data derived from the hits in a database comprises computer readable program code for transferring the temporarily stored traffic data to the database.

48. The article of manufacture of claim 47, wherein the computer readable program code for transferring the temporarily stored traffic data to the database sorts the temporarily stored traffic data before transferring the traffic data to the database.

49. The article of manufacture of claim 40, wherein the computer readable program code for processing hits logged in each log file further comprises computer readable program code for reading log lines in the log files, and for separating each log line into individual fields.

50. The article of manufacture of claim 49, further comprising computer readable program code for generating reports using the associated traffic data stored in the database.

51. The article of manufacture of claim 40, wherein the computer readable program code for processing hits logged in each log file processes hits originating from multiple internet sites and logged to a single log file.

52. The article of manufacture of claim 40, wherein the computer readable program code for processing hits logged in each log file comprises computer readable program code for identifying a source of each hit.

53. The article of manufacture of claim 40, wherein the computer readable program code for processing hits logged in each log file comprises computer readable program code for determining host and domain information for each visitor.

54. The article of manufacture of claim 53, wherein the DNS resolver means computer readable program code for determining host and domain information for each visitor utilizes reverse DNS resolution to determine the host and domain information for each visitor.

55. A system for analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, comprising:

a database;

a log engine that receives log files from the at least one internet server, processes hits logged in each log file, and stores traffic data extracted from the processed hits in the database, wherein the log engine comprises,

a database buffer that temporarily stores traffic data received from the database,

a log parser that processes each hit in each log file, and separates each hit into its individual fields, wherein the log parser, when new log file data is present in the log file, processes said new log file data and determines an “

end of file”

location on the log file, and, when new log file data is not present in the log file, periodically checks the log file at predetermined time periods to check for new log file data, and commences processing of any new log file data at a most recent determined “

end of file”

location.

a visitor identifier that receives each hit'"'"'s individual fields from the log parser, identifies each hit as originating from either a new visitor or an existing visitor, and creates a new visitor record in the database buffer if a hit originates from a new visitor,

a buffer updater that, prior to processing a new log file, copies previously stored data from the database to the database buffer, and wherein, for each hit, the buffer updater locates in the database buffer the visitor record identified or created by the visitor identifier for a respective hit, and updates the identified or created visitor record in the database buffer with traffic data derived from the respective hit, and

a database updater that copies updated traffic data from the database buffer to the database after all hits in the new log file have been processed; and

a report engine that generates reports using the traffic data stored in the database.

56. The system of claim 55, wherein the log engine further comprises a website identifier that identifies a source of each hit.

57. The system of claim 56, wherein the website identifier identifies the source of each hit from website identifier text received from the log parser for each hit.

58. The system of claim 55, wherein the log engine further comprises a domain name system (DNS) resolver that determines host and domain information for each visitor to an internet site.

59. The system of claim 58, wherein the DNS resolver is adapted to process multiple DNS queries in parallel.

60. The system of claim 55, wherein the log engine is adapted to process e-commerce log files that contain information on money spent by the visitor.

61. A method of analyzing and monitoring internet traffic generated by visitors to at least one internet site hosted by at least one internet server, comprising the steps of:

receiving log files from the at least one internet server;

processing hits logged in each log file, as each hit is logged to each log file, by;

(a) processing new hits present in a log file,

(b) determining an “

end of file”

location on the log file,

(c) waiting for a predetermined time period if no new hits are present in the log file,

(d) checking for new hits in the log file after the predetermined time period, and

(e) processing any new hits discovered in the log file by starting at the determined end of file location in the log file;

storing, in a database, traffic data derived from the hits; and

associating the traffic data derived from the hits and stored in the database with a visitor that generated the hit.

62. The method of claim 61, further comprising the step of generating reports using the associated traffic data stored in the database.

63. The method of claim 62, wherein the traffic data derived from the hits is stored in a plurality of hash tables.

64. The method of claim 63, wherein traffic information derived from the hits are stored in a visitor hash table that contains a unique visitor record for each visitor, and wherein data related to at least one non-unique parameter is stored in respective data tables.

65. The method of claim 64, wherein the visitor hash table comprises at least one pointer that points to at least one record stored in at least one of the data tables.

66. The method of claim 64, further comprising the steps of:

determining if a hit originates from a visitor with a preexisting visitor record in the database; and

creating a new visitor record if the hit originates from a visitor without a preexisting visitor record in the database.

67. The method of claim 61, further comprising the step of temporarily storing the traffic data derived from the hits in a buffer prior to storing the traffic data in the database.

68. The method of claim 67, further comprising the step of sorting the traffic data stored in the buffer prior to storing the traffic data in the database.

69. The method of claim 61, wherein the step of processing hits logged in each log file comprises the steps of:

reading log lines in the log files; and

separating each log line into individual fields.

70. The method of claim 69, wherein the hits logged in each log file are processed in real time as each hit is logged to a log file.

71. The method of claim 61, further comprising the step of identifying a source from which each hit originates.

72. The method of claim 61, further comprising the step of determining host and domain information for each visitor.

73. The method of claim 72, wherein host and domain information for each visitor is determined using reverse domain name system (DNS) resolution.

74. A method of processing a log file to obtain traffic data, comprising the steps of:

copying previously stored traffic data from a database to a database buffer;

separating hits logged in the log file into individual fields, wherein each hit is processed as it is logged to the log file by;

(a) processing new hits present in the log file,

(b) determining an “

end of file”

location in the log file,

(c) waiting for a predetermined time period if no new hits are present in the log file,

(d) checking for new hits in the log file after the predetermined time period,

(e) processing any new hits discovered in the log file by starting at the end of file location in the log file, and

(f) repeating steps (a)-(e);

identifying each hit as originating from either a new visitor or an existing visitor;

creating a new visitor record in the database buffer if a hit originates from a new visitor;

for each hit, locating the visitor record identified or created and updating the identified or created visitor record in the database buffer with traffic data derived from the respective hit; and

copying updated traffic data from the database buffer to the database after all hits in the log file have been processed.

75. The method of claim 74, further comprising the step of generating a report based on the traffic data in the database.

76. The method of claim 74, wherein hits originating from multiple sources are logged to the log file.

77. The method of claim 76, further comprising the step of identifying a source from which each hit originates.

78. The method of claim 74, further comprising the step of determining host and domain information for each visitor.

79. The method of claim 78, wherein host and domain information for each visitor is determined using reverse domain name system (DNS) resolution.