Variable sampling rates for website visitation analysis

• US 8,578,041 B2
• Filed: 12/27/2005
• Issued: 11/05/2013
• Est. Priority Date: 06/03/2005
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Error, Christopher Reid
• Primary Examiner(s): Fan, Hua
• Assistant Examiner(s): Nguyen, Van Kim T
• Application Number:
  US11/319,337
• Publication Number:
  US 20060274763A1
• Time in Patent Office: 2,870 Days
• Field of Search: 709/231, 709/223, 709/229
• US Class Current: 709/231
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A data set containing website traffic data or other data is sampled according to a variable sample rate. A target number of samples per time period is established, and a baseline sample rate is determined. Data items in the data set are sampled according to the baseline sample rate, to obtain a sample set. For time periods where the size of the resulting sample set exceeds the target number of samples, a new sample rate is established and the data items for the time period are resampled. Appropriate sampling capability can thus be provided for website traffic in normal time periods, while maintaining capability for handling spikes and other variations in website traffic as may take place in response to certain periodic or non-periodic events.

Claims

1. A method for sampling a data set comprising a plurality of data items, comprising:

establishing a target number of sample items to be extracted from a plurality of data items corresponding to a time period;

establishing a tolerance factor;

establishing a target range using the target number and the tolerance factor;

establishing a first sampling rate for sampling the plurality of data items;

applying, by a computer, the first sampling rate to the plurality of data items corresponding to the time period to obtain a first sample set comprising a first plurality of sample items, wherein the first sample set is a subset of the plurality of data items;

determining, by the computer, a number of sample items contained in the first sample set; and

responsive to the number of sample items contained in the first sample set being substantially different from the target number for the time period falling outside the target range;

establishing a second sampling rate that is different from the first sampling rate; and

applying, by the computer, the second sample sampling rate to the plurality of data items to obtain a second sample set comprising a second plurality of sample items.

2. The method of claim 1, wherein obtaining a applying the second sampling rate to obtain the second sample set comprises applying the second sampling rate to the plurality of data items corresponding to the time period.

3. The method of claim 1, wherein obtaining a applying the second sampling rate to obtain the second sample set comprises:

establishing a resampling rate based on the second sampling rate and the first sampling rate; and

applying the resampling rate to the first sample set.

4. The method of claim 3, wherein establishing the resampling rate comprises dividing the second sampling rate by the first sampling rate.

5. The method of claim 1, further comprising generating a report using at least one of the first and second sample sets.

6. The method of claim 1, wherein the plurality of data items describe website visits.

7. The method of claim 6, further comprising generating a website traffic report using at least one of the first and second sample sets.

8. The method of claim 1, wherein the time period comprises a day.

9. The method of claim 1, further comprising:

applying the first sampling rate to the plurality of data items corresponding to a second time period to obtain an third sample set comprising a third plurality of sample items; and

responsive to the third sample set containing an additional number of sample items for the second time period substantially different from the target number falling outside the target range, performing the steps of;

establishing a third sampling rate; and

obtaining a fourth sample set using the third sampling rate.

10. The method of claim 1, further comprising:

applying the second sampling rate to the plurality of data items corresponding to a second time period to obtain a third sample set comprising a third plurality of sample items; and

responsive to the third sample set containing an additional number of sample items for the second time period substantially different from the target number falling outside the target range, performing the steps of;

establishing a third sampling rate; and

obtaining a fourth sample set using the third sampling rate.

11. The method of claim 1, wherein establishing a target number the target range of sample items comprises establishing a target number the target range based on at least one selected from the group consisting of performance, traffic, load factors, storage space, and reporting needs.

12. The method of claim 1, wherein establishing a target number the target range of sample items comprises receiving a target number the target range.

13. The method of claim 1, wherein establishing the first sampling rate comprises establishing the first sampling rate based on the target number range of sample items for the time period and an expected number of data items for the time period.

14. The method of claim 1, wherein establishing the first sampling rate comprises receiving the first sampling rate.

15. A method for sampling a data set comprising a plurality of data items, each data item associated with a time period, comprising:

establishing a target number of sample items to be extracted from a plurality of data items corresponding to one or more time periods;

establishing a tolerance factor;

establishing a target range using the target number and the tolerance factor;

establishing a first sampling rate for sampling the plurality of data items;

for each of the one or more time periods;

applying, by a computer, the first sampling rate to the plurality of data items corresponding to the time period to obtain a first sample set comprising a first plurality of sample items for a given time period of the one or more time periods, wherein the first sample set is a subset of the plurality of data items corresponding to the particular given time period; and

determining, by the computer, a number of sample items contained in the first sample set;

responsive to the number of sample items contained in the first sample set for the time period being substantially different from the target number for the given time period falling outside the target range;

establishing a second sampling rate for the given time period, wherein the second sampling rate is different from the first sampling rate; and

applying, by the computer, the second sample sampling rate to the plurality of data items corresponding to the given time period to obtain a second sample set for the given time period using the second sampling rate, wherein the second sample set comprises a second plurality of sample items.

16. A computer program product for sampling a data set comprising a plurality of data items, comprising:

a non-transitory computer-readable storage medium; and

computer program code, encoded on the non-transitory computer-readable storage medium, for;

establishing a target range of sample items to be extracted from a plurality of data items corresponding to a time period;

establishing a tolerance factor;

establishing a target range using the target number and the tolerance factor;

establishing a first sampling rate for sampling the plurality of data items;

applying the first sampling rate to the plurality of data items corresponding to the time period to obtain a first sample set comprising a first plurality of sample items, wherein the first sample set is a subset of the plurality of data items;

determining a number of sample items contained in the first sample set; and

responsive to the number of sample items contained in the first sample set being substantially different from the target number for the time period falling outside the target range;

establishing a second sampling rate that is different from the first sampling rate; and

applying the second sample sampling rate to the plurality of data items to obtain a second sample set comprising a second plurality of sample items.

17. The computer program product of claim 16, wherein the computer program code for obtaining a applying the second sampling rate to obtain the second sample set comprises computer program code for applying the second sampling rate to the plurality of data items corresponding to the time period.

18. The computer program product of claim 16, wherein the computer program code for obtaining a applying the second sampling rate to obtain the second sample set comprises computer program code for:

establishing a resampling rate based on the second sampling rate and the first sampling rate; and

applying the resampling rate to the first sample set.

19. The computer program product of claim 18, wherein the computer program code for establishing the resampling rate comprises computer program code for dividing the second sampling rate by the first sampling rate.

20. The computer program product of claim 16, further comprising computer program code for generating a report using at least one of the first and second sample sets.

21. The computer program product of claim 16, wherein the plurality of data items describe website visits.

22. The computer program product of claim 21, further comprising computer program code for generating a website traffic report using at least one of the first and second sample sets.

23. The computer program product of claim 16, wherein the time period comprises a day.

24. The computer program product of claim 16, further comprising computer program code for:

applying the first sampling rate to the plurality of data items corresponding to a second time period to obtain an third sample set comprising a third plurality of sample items; and

responsive to the third sample set containing an additional number of sample items for the second time period substantially different from the target number falling outside the target range;

establishing a third sampling rate; and

obtaining a fourth sample set using the third sampling rate.

25. The computer program product of claim 16, further comprising computer program code for:

applying the second sampling rate to the plurality of data items corresponding to a second time period to obtain an third sample set comprising a third plurality of sample items; and

responsive to the third sample set containing an additional number of sample items for the second time period substantially different from the target number falling outside the target range;

establishing a third sampling rate; and

obtaining a fourth sample set using the third sampling rate.

26. The computer program product of claim 16, wherein the computer program code for establishing a target number the target range of sample items comprises computer program code for establishing a target number the target range based on at least one selected from the group consisting of performance, traffic, load factors, storage space, and reporting needs.

27. The computer program product of claim 16, wherein the computer program code for establishing a target number the target range of sample items comprises computer program code for receiving a target number the target range.

28. The computer program product of claim 16, wherein the computer program code for establishing the first sampling rate comprises computer program code for establishing the first sampling rate based on the target number range of sample items for the time period and an expected number of data items for the time period.

29. The computer program product of claim 16, wherein the computer program code for establishing the first sampling rate comprises computer program code for receiving the first sampling rate.

30. A computer system for sampling a data set comprising a plurality of data items, comprising:

a processor configured to execute instructions embodied in a storage device and comprising a log processing module configured for;

establishing a target number of sample items to be extracted from a plurality of data items corresponding to a time period,

establishing a tolerance factor;

establishing a target range using the target number and the tolerance factor;

establishing a first sampling rate for sampling the plurality of data items;

applying the first sampling rate to the plurality of data items corresponding to the time period to obtain a first sample set comprising a first plurality of sample items, wherein the first sample set is a subset of the plurality of data items;

determining a number of sample items contained in the first sample set; and

responsive to the number of sample items contained in the first sample set being substantially different from the target number falling outside the target range;

establishing a second sampling rate that is different from the first sampling rate; and

applying the second sample sampling rate to the plurality of data items to obtain a second sample set comprising a second plurality of sample items; and

wherein the storage device is configured for storing at least one of the first and second sample sets.

31. The system of claim 30, wherein the log processing module obtains a is configured for applying the second sampling rate to the plurality of data items to obtain the second sample set by applying the second sampling rate to the plurality of data items corresponding to the time period.

32. The system of claim 30, wherein the log processing module obtains a is configured for applying the second sampling rate to the plurality of data items to obtain the second sample set by:

establishing a resampling rate based on the second sampling rate and the first sampling rate; and

applying the resampling rate to the first sample set.

33. The system of claim 32, wherein the log processing module establishes a is configured for establishing the resampling rate by dividing the second sampling rate by the first sampling rate.

34. The system of claim 30, further comprising a report generator module configured for generating a report using at least one of the first and second sample sets.

35. The system of claim 30, wherein the plurality of data items describe website visits.

36. The system of claim 35, further comprising a report generator, for generating a website traffic report using at least one of the first and second sample sets.

37. The system of claim 30, wherein the time period comprises a day.

38. The system of claim 30, wherein the log processing module is further configured for:

the log processing module applies applying the first sampling rate to the plurality of data items corresponding to a second time period to obtain an third sample set comprising a third plurality of sample items; and

responsive to the third sample set containing an additional number of sample items for the second time period substantially different from the target number, the log processing module falling outside the target range;

establishes establishing a third sampling rate; and

obtains obtaining a fourth sample set using the third sampling rate.

39. The system of claim 30, wherein the log processing module is further configured for:

the log processing module applies applying the second sampling rate to the plurality of data items corresponding to a second time period to obtain an third sample set comprising a third plurality of sample items; and

responsive to the third sample set containing an additional number of sample items for the second time period substantially different from the target number, the log processing module falling outside the target range;

establishes establishing a third sampling rate; and

obtains obtaining a fourth sample set using the third sampling rate.

40. The system of claim 30, wherein the log processing module establishes a target number is configured for establishing the target range of sample items by establishing a target number the target range based on at least one selected from the group consisting of performance, traffic, load factors, storage space, and reporting needs.

41. The system of claim 30, wherein the log processing module establishes a target number is configured for establishing the target range of sample items by receiving a target number the target range.

42. The system of claim 30, wherein the log processing module establishes a target number is configured for establishing the first sampling rate by establishing the first sampling rate based on the target number range of sample items for the time period and an expected number of data items for the time period.

43. The system of claim 30, wherein the log processing module establishes a target number is configured for establishing the first sampling rate by receiving the first sampling rate.