METHOD AND SYSTEM FOR DISTRIBUTED PROCESSING OF WEB TRAFFIC ANALYTICS DATA

• US 20110225287A1
• Filed: 03/12/2010
• Published: 09/15/2011
• Est. Priority Date: 03/12/2010
• Status: Abandoned Application

• Current Assignee: WebTrends Incorporated (WebTrends Holding, Inc.)
• Original Assignee: WebTrends Incorporated (WebTrends Holding, Inc.)
• Inventors: EASTERDAY, JOHN L., DALAL, MUKESH
• Application Number:
  US12/723,478
• Publication Number:
  US 20110225287A1
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A distributed analytics method and system for processing hit data and producing reports. The hit data includes attributes and values representing activities of a visitor on a web site. The distributed analytics system may include log processors to read and parse the hit data. Analytics generators are configured to read and process the parsed hit data and store the processed data in one or more analytics data stores, which are associated with bands. Analytics processors read and merge data from the analytics data stores associated with the bands, and produce intermediate report deltas. Report generators are configured to receive the intermediate report deltas and produce and store one or more report segments in report data stores. External data can be combined with the data being processed. Report processors are configured to read data from the report data stores and produce a final result based on the report segments.

Claims

1. A method for distributed processing of analytics data, comprising:

generating a plurality of analytics data store files, each analytics data store file including analytics information corresponding to a predefined period of time and stored in one of a plurality of bands;

configuring a plurality of analytics generators, each analytics generator being associated with one of the plurality of bands;

a first analytics generator reading and processing event data associated with a first individual; and

a second analytics generator reading and processing event data associated with a second individual,

wherein the first individual is associated with one of the plurality of bands, and the second individual is associated with another of the plurality of bands.

2. The method of claim 1, further comprising:

the first analytics generator reading historical event data from a recent historical analytics data store file associated with the first individual, and generating a first new analytics data store file including at least some of the associated event data and historical event data; and

the second analytics generator reading historical event data from a recent historical analytics data store file associated with the second individual, and generating a second new analytics data store file including at least some of the associated event data and historical event data.

3. The method of claim 2, wherein the first new analytics data store file is stored in one of the plurality of bands, and the second new analytics data store file is stored in another of the plurality of bands.

4. The method of claim 2, further comprising:

configuring a plurality of analytics processors, each of the plurality of bands being associated with one or more of the analytics processors; and

each of the analytics processors reading and merging data from one or more of the analytics data store files, and producing one or more intermediate report deltas.

5. The method of claim 4, further comprising:

configuring a plurality of report generators, the report generators receiving the one or more intermediate report deltas.

6. The method of claim 5, further comprising:

outputting data from the one or more intermediate report deltas to one or more data consumers.

7. The method of claim 5, further comprising:

merging data from the one or more intermediate report deltas into one or more report segments; and

storing the one or more report segments in corresponding one or more report data stores.

8. The method of claim 7, further comprising:

configuring a plurality of report processors, each report processor reading data from the one or more report data stores and producing a final result based on the report segments.

9. The method of claim 8, wherein the final result is based on at least two merged report segments.

10. The method of claim 8, wherein the final result is based on individual report segments.

11. The method of claim 8, wherein the final result includes a top N determination based on predefined criteria.

12. The method of claim 8, further comprising:

the report processors transmitting the final result to one or more data consumers.

13. The method of claim 8, further comprising:

a slave report processor producing a portion of the final result; and

a master report processor marshaling the slave report processor, combining the portion of the final result produced by the slave report processor with another portion of the final result, and outputting the final result.

14. The method of claim 1, further comprising:

combining external data with the event data associated with the first individual or the second individual.

15. The method of claim 4, further comprising:

combining external data with data from one or more of the analytics data store files, and producing the one or more intermediate report deltas based in part on the external data.

16. The method of claim 8, further comprising:

combining external data with data from the one or more report data stores and producing the final result based in part on the external data.

17. A web-traffic analytics device, comprising:

an analytics generator configured to read event data associated with a web site visitor, and to store the event data in an analytics data store associated with a band;

an analytics processor associated with the band, the analytics processor being configured to read and merge data from the analytics data store, and to produce an intermediate report delta;

a report generator configured to receive the intermediate report delta from the analytics processor, and to partition data from the intermediate report delta into one or more report segments;

a report data store to store the one or more report segments; and

a report processor to read data from the report data store and to produce a final result based on the one or more report segments.

18. The web-traffic analytics device of claim 17, further comprising:

a log data store to receive hit data and to store the hit data in the band; and

a log processor to examine the hit data, and to parse a visitor identification (ID) and associated event attributes, wherein the analytics generator is configured to receive the visitor ID and the event attributes from the log processor and generate the analytics data store associated with the band.

19. A distributed analytics system, comprising:

a log data store to receive one or more hits and to store the one or more hits in one or more first bands, wherein each hit represents activities of a visitor on a web site;

one or more log processors to examine each hit within the one or more first bands, and to parse a visitor identification (ID) and associated plurality of event attributes; and

one or more analytics generators to receive the visitor ID and the plurality of event attributes from the one or more log processors, and to generate at least one analytics data file to be stored in one or more second bands,

wherein each of the second bands is associated with a corresponding one of the first bands.

20. The distributed analytics system of claim 19, wherein each of the bands is associated with one computer server.

21. The distributed analytics system of claim 19, wherein the one or more first bands is associated with a first computer server, and the one or more second bands is associated with a second computer server.

22. The distributed analytics system of claim 19, wherein a visitor is assigned to the one or more first bands based on a function of (visitor ID modulo L), where L is the number of first bands.

23. The distributed analytics system of claim 19, wherein a visitor is assigned to the one or more first bands based on a function of geographic location of the visitor.

24. The distributed analytics system of claim 23, further comprising:

one or more analytics data stores to store the at least one analytics data file in the one or more second bands; and

one or more analytics processors to merge the at least one analytics data file with at least one second analytics data file and to produce an intermediate report delta.

25. The distributed analytics system of claim 24, wherein more than one analytics processor is associated with a single one of the second bands.

26. The distributed analytics system of claim 24, further comprising:

one or more report generators operatively associated with the one or more analytics processors, the one or more report generators configured to receive the intermediate report delta.

27. The distributed analytics system of claim 26, wherein the one or more report generators is configured to output data from the intermediate report delta to one or more data consumers.

28. The distributed analytics system of claim 26, wherein the one or more report generators is configured to merge data from the intermediate report delta into one or more report segments.

29. The distributed analytics system of claim 28, further comprising:

one or more report data stores associated with the one or more report segments.

30. The distributed analytics system of claim 29, further comprising:

one or more report processors for producing a final result based on the report segments.

31. The distributed analytics system of claim 30, wherein more than one report processor is associated with a single one of the report data stores.

32. The distributed analytics system of claim 30, wherein the final result comprises a single combined final result based on at least two merged report segments.

33. The distributed analytics system of claim 30, wherein:

the final result comprises individual portions of the final result that are based on individual report segments; and

each of the report processors is configured to transmit a corresponding portion of the final result to a data consumer over an individual channel.

34. The distributed analytics system of claim 30, wherein the one or more report processors is configured to transmit the final result to one or more data consumers.

35. The distributed analytics system of claim 30, wherein the one or more report processors comprise:

a slave report processor to produce a portion of the final result; and

a master report processor configured to marshal the slave report processor, to combine the portion of the final result produced by the slave report processor with another portion of the final result, and to output the final result.

36. A method for redistributing analytics data in a distributed analytics system, comprising:

configuring a first band for storing web traffic analytics data;

processing the web traffic analytics data using the first band;

configuring a second band in the distributed analytics system; and

processing the web traffic analytics data using the first and second bands.

37. The method of claim 36, further comprising:

instantiating a first analytics generator to process the web traffic analytics data when the first band is configured; and

after configuring the second band, instantiating a second analytics generator to process a range of the web traffic analytics data, and configuring the first analytics generator to process a different range of the web traffic analytics data.

38. The method of claim 36, further comprising:

redistributing all of the web traffic analytics data stored in the second band to the first band; and

removing the second band from the distributed analytics system.

39. The method of claim 36, wherein the first band is associated with a first computer server and the second band is associated with a second different computer server.