Apparatus and methods for classification of web sites

• US 20040111504A1
• Filed: 12/10/2002
• Published: 06/10/2004
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Yu, Shun-Zheng, Xia, Honghui, Liu, Zhen, Zhang, Li, Halim, Nagui, Squillante, Mark Steven
• Granted Patent: US 7,792,951 B2
• Time in Patent Office: Days
• US Class Current: 709/223
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

Apparatus and methods for classifying web sites are provided. With the apparatus and methods, traffic data is obtained for a plurality of web sites. This patterns, or templates, for each web site are generated based on this traffic data and the patterns are clustered into classes of web sites using a clustering algorithm. The clusters, or classes, are then profiled to generate a template for each class. The template for each class is generated by first shifting the patterns for each web site that is part of the class to compensate for effects like time zone differences, if any, and then identifying a pattern that is most similar to all of the patterns in the class. Once the template for each class is generated, this template is then used with traffic data from a new web site to classify the new web site into one of the existing classes. In other words, when traffic data for a new web site is received, a pattern for the traffic data of the new web site is generated and compared to the templates for the various classes. If a matching class template is identified, the new web site is classified into the corresponding class. If the pattern for the new web site does not match any of the existing templates, a new template and class may be generated based on the pattern for the new web site.

Claims

1. A computer program product in a computer readable medium for characterizing a web site, comprising:

first instructions for receiving traffic data for the web site;

second instructions for matching the traffic data to a matching traffic template of a plurality of predefined traffic templates; and

third instructions for characterizing the web site based on characteristics of the matching traffic template.

2. The computer program product of claim 1, further comprising:

fourth instructions for receiving traffic data for a plurality of web sites;

fifth instructions for clustering the plurality of web sites into two or more clusters based on the traffic data; and

sixth instructions for profiling the two or more clusters based on traffic data for member web sites of each cluster, to generate the plurality of predefined traffic templates.

3. The computer program product of claim 2, wherein the fifth instructions for clustering include instructions for using a nearest neighbor clustering algorithm on traffic patterns of the traffic data for the plurality of web sites.

4. The computer program product of claim 2, wherein the fifth instructions for clustering the plurality of web sites into two or more clusters based on the traffic data include:

instructions for identifying a traffic pattern for each web site in the plurality of web sites; and

instructions for clustering the plurality of web sites based on similarities between the traffic pattern for each web site and each other web site in the plurality of web sites.

5. The computer program product of claim 4, wherein the traffic pattern is a normalized weighted average request pattern.

6. The computer program product of claim 4, wherein the fifth instructions for clustering the plurality of web sites into two or more clusters further include instructions for shifting one or more of the traffic patterns for each of the web sites in the plurality of web sites to compensate for differences in time zones.

7. The computer program product of claim 4, wherein the fifth instructions for clustering the plurality of web sites into two or more clusters further include:

calculating a dissimilarity measure between each pair of web sites in the plurality of web sites; and

clustering the plurality of web sites based on the dissimilarity measures for each pair of web sites.

8. The computer program product of claim 2, wherein the sixth instructions for profiling the two or more clusters include:

instructions for identifying, for each cluster, a traffic template that is most similar to traffic patterns of all of the member web sites of the cluster.

9. The computer program product of claim 8, wherein the traffic template that is most similar to traffic patterns of all of the member web sites of the cluster is identified by averaging the traffic patterns of all of the member web sites of the cluster.

10. The computer program product of claim 8, wherein the traffic template that is most similar to traffic patterns of all of the member web sites of the cluster is identified by minimizing a maximum difference between the traffic template and each traffic pattern of each member web site of the cluster.

11. The computer program product of claim 1, wherein the second instructions for matching the traffic data to a matching traffic template of a plurality of predefined traffic templates include:

instructions for determining if the traffic data does not match any of the plurality of predefined traffic templates; and

instructions for generating a new traffic template based on the traffic data if the traffic data does not match any of the plurality of predefined traffic templates.

12. The computer program product of claim 1, further comprising:

fourth instructions for performing at least one of traffic prediction, capacity planning and hot-spot detection based on the characterizing of the web site.

13. The computer program product of claim 2, further comprising:

seventh instructions for identifying characteristics of the member web sites of each cluster; and

eighth instructions for identifying characteristics of the cluster based on the characteristics of the member web sites of the cluster.

14. A method of characterizing a web site, comprising:

receiving traffic data for the web site;

matching the traffic data to a matching traffic template of a plurality of predefined traffic templates; and

characterizing the web site based on characteristics of the matching traffic template.

15. The method of claim 14, further comprising:

receiving traffic data for a plurality of web sites;

clustering the plurality of web sites into two or more clusters based on the traffic data; and

profiling the two or more clusters based on traffic data for member web sites of each cluster, to generate the plurality of predefined traffic templates.

16. The method of claim 15, wherein clustering includes using a nearest neighbor clustering algorithm on traffic patterns for the traffic data for the plurality of web sites.

17. The method of claim 15, wherein clustering the plurality of web sites into two or more clusters based on the traffic data includes:

identifying a traffic pattern for each web site in the plurality of web sites; and

clustering the plurality of web sites based on similarities between the traffic pattern for each web site and the traffic pattern for each other web site in the plurality of web sites.

18. The method of claim 17, wherein the traffic pattern is a normalized weighted average request pattern.

19. The method of claim 17, wherein clustering the plurality of web sites into two or more clusters further includes shifting one or more of the traffic patterns for each of the web sites in the plurality of web sites to compensate for differences in time zones.

20. The method of claim 17, wherein clustering the plurality of web sites into two or more clusters further includes:

calculating a dissimilarity measure between each pair of web sites in the plurality of web sites; and

clustering the plurality of web sites based on the dissimilarity measures for each pair of web sites.

21. The method of claim 15, wherein profiling the two or more clusters includes:

identifying, for each cluster, a traffic template that is most similar to traffic patterns of all of the member web sites of the cluster.

22. The method of claim 21, wherein the traffic template that is most similar to traffic patterns of all of the member web sites of the cluster is identified by averaging the traffic patterns of all of the member web sites of the cluster.

23. The method of claim 21, wherein the traffic template that is most similar to traffic patterns of all of the member web sites of the cluster is identified by minimizing a maximum difference between the traffic template and each traffic pattern of each member web site of the cluster.

24. The method of claim 14, wherein matching the traffic data to a matching traffic template of a plurality of predefined traffic templates includes:

determining if the traffic data does not match any of the plurality of predefined traffic templates; and

generating a new traffic template based on the traffic data if the traffic data does not match any of the plurality of predefined traffic templates.

25. The method of claim 14, further comprising:

performing at least one of traffic prediction, capacity planning and hot-spot detection based on the characterizing of the web site.

26. The method of claim 15, further comprising:

identifying characteristics of the member web sites of each cluster; and

identifying characteristics of the cluster based on the characteristics of the member web sites of the cluster.

27. An apparatus for characterizing a web site, comprising:

means for receiving traffic data, for the web site;

means for matching the traffic data to a matching traffic template of a plurality of predefined traffic templates; and

means for characterizing the web site based on characteristics of the matching traffic template.

28. The apparatus of claim 27, further comprising:

means for receiving traffic data for a plurality of web sites;

means for clustering the plurality of web sites into two or more clusters based on the traffic data; and

means for profiling the two or more clusters based on traffic data for member web sites of each cluster, to generate the plurality of predefined traffic templates.

29. The apparatus of claim 28, wherein the means for clustering includes means for using a nearest neighbor clustering algorithm on traffic patterns for the traffic data for the plurality of web sites.

30. The apparatus of claim 28, wherein the means for clustering the plurality of web sites into two or more clusters based on the traffic data includes:

means for identifying a traffic pattern for each web site in the plurality of web sites; and

means for clustering the plurality of web sites based on similarities between the traffic pattern for each web site and the traffic pattern for each other web site in the plurality of web sites.

31. The apparatus of claim 30, wherein the traffic pattern is a normalized weighted average request pattern.

32. The apparatus of claim 30, wherein the means for clustering the plurality of web sites into two or more clusters further includes means for shifting one or more of the traffic patterns for each of the web sites in the plurality of web sites to compensate for differences in time zones.

33. The apparatus of claim 30, wherein the means for clustering the plurality of web sites into two or more clusters further includes:

means for calculating a dissimilarity measure between each pair of web sites in the plurality of web sites; and

means for clustering the plurality of web sites based on the dissimilarity measures for each pair of web sites.

34. The apparatus of claim 30, wherein the means for profiling the two or more clusters includes:

means for identifying, for each cluster, a traffic template that is most similar to traffic patterns of all of the member web sites of the cluster.

35. The apparatus of claim 34, wherein the traffic template that is most similar to traffic patterns of all of the member web sites of the cluster is identified by averaging the traffic patterns of all of the member web sites of the cluster.

36. The apparatus of claim 34, wherein the traffic template that is most similar to traffic patterns of all of the member web sites of the cluster is identified by minimizing a maximum difference between the traffic template and each traffic pattern of each member web site of the cluster.

37. The apparatus of claim 27, wherein the means for matching the traffic data to a matching traffic template of a plurality of predefined traffic templates includes:

means for determining if the traffic data does not match any of the plurality of predefined traffic templates; and

means for generating a new traffic template based on the traffic data if the traffic data does not match any of the plurality of predefined traffic templates.

38. The apparatus of claim 27, further comprising:

means for performing at least one of traffic prediction, capacity planning and hot-spot detection based on the characterizing of the web site.

39. The apparatus of claim 28, further comprising:

means for identifying characteristics of the member web sites of each cluster; and

means for identifying characteristics of the cluster based on the characteristics of the member web sites of the cluster.

40. A method for determining characteristics of one or more web sites based on traffic data, comprising:

receiving traffic data for a plurality of web sites;

clustering the plurality of web sites into two or more clusters based on the traffic data;

profiling the two or more clusters based on traffic data for member web sites of each cluster, to generate a traffic template for each cluster;

identifying characteristics of the member web sites of each cluster; and

identifying characteristics of the cluster based on the characteristics of the member web sites of the cluster.

41. A method for deploying computing infrastructure, comprising integrating computer readable code into a computing system, wherein the code in combination with the computing system is capable of performing the following:

receiving traffic data for the web site;

matching the traffic data to a matching traffic template of a plurality of predefined traffic templates; and

characterizing the web site based on characteristics of the matching traffic template.