Method and apparatus for generating time-series data from web pages

• US 7,526,462 B2
• Filed: 03/16/2006
• Issued: 04/28/2009
• Est. Priority Date: 05/26/2005
• Status: Expired due to Fees

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Sawa, Norihiko, Sakurai, Shigeaki
• Primary Examiner(s): Vincent; David R
• Assistant Examiner(s): Tran; Mai T
• Application Number:
  US11/376,876
• Publication Number:
  US 20060271533A1
• Time in Patent Office: 1,139 Days
• Field of Search: 706/20, 707/6
• US Class Current: 706/20
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

According to one embodiment, the Web pages that match a user'"'"'s designated collection condition are collected from a plurality of Web sites. The collected Web pages are divided into a plurality of clusters, based on URL information of the Web pages. A date expression is extracted from Web pages included in each of the clusters. A typical date expression form is determined for each of the clusters, based on the extracted date expression. The Web pages included in each of the clusters are divided into a plurality of items, based on the date expression form. The items are sorted for each of the clusters in order of time, based on date expressions corresponding to the items. Time-series data is generated for each of the clusters by sorting the items.

Claims

1. A computer-implemented method of generating time-series data from Web pages, comprising:

collecting Web pages, which match a user'"'"'s designated collection condition, from a plurality of Web sites, the collecting including storing the collected Web pages in a storage device;

dividing a set of Web pages stored in the storage device into a plurality of clusters, based on URL information of the Web pages;

extracting a date expression from Web pages included in each of the clusters;

determining a typical date expression form for each of the clusters, based on the extracted date expression;

dividing the Web pages included in each of the clusters into a plurality of items with reference to a location where a date expression of the date expression form appears, based on the date expression form;

generating time-series data for each of the clusters by sorting the items for each of the clusters in order of time, based on date expressions corresponding to the items;

providing the Web pages stored in the storage device with URL features, which represent features of URL information of the Web pages, as features of the Web pages, the providing including extracting the URL features by dividing the URL information; and

generating feature vectors, which represent the URL features provided for the Web pages, as URL feature vectors, based on the URL features,

wherein;

the set of Web pages corresponding to the URL feature vector of each of the Web pages is divided into a plurality of clusters, based on the URL feature vector;

the URL features each include part of URL information from which each of the URL features is extracted, as an attribute;

the URL feature vectors have all attributes of the URL features in common without any, redundancy; and

the providing includes;

dividing the URL information into a plurality of divided character strings by a plurality of delimiters; and

determining each of the divided character strings as one of an attribute and an attribute value of each of the URL features, the determining including setting each of the divided character strings to one of an attribute having presence or absence of a character string as an attribute value, an attribute having a divided character string subsequent to one of the delimiters as an attribute value, and an attribute value having a divided character string precedent to one of the delimiters as an attribute, in accordance with types of the delimiters, and wherein;

the delimiters are classified into two types of a first delimiter and a second delimiter;

the dividing the URL information includes dividing the URL information into a plurality of divided character strings by the first delimiter, and dividing some of the divided character strings, which include the second delimiter, into a pair of divided character strings by the second delimiter; and

the determining each of the divided character strings includes determining the divided character strings, which are obtained by the first delimiter and exclude the second delimiter, as an attribute, determining presence or absence of the character strings as an attribute value of the attribute, determining one of the divided character strings obtained by the second delimiter, which is precedent to the second delimiter, as an attribute, and determining other of the divided character strings, which is subsequent to the second delimiter, as an attribute value.

2. The method according to claim 1, further comprising:

computing a degree of similarity between the clusters, based on text information of Web pages included in the clusters;

integrating clusters whose degrees of similarity exceed a given threshold value into a new cluster; and

repeating the computing and the integrating until clusters whose degrees of similarity exceed the given threshold value are completely processed.

3. The method according to claim 2, further comprising:

extracting an expression that features each of the clusters from text information of Web pages included in each of the clusters; and

generating a text feature vector that represents a feature of each of the clusters, based on the extracted expression,

wherein, in the computing, a degree of similarity between the clusters is computed based on the text feature vector.

4. The method according to claim 3, further comprising acquiring an appearance ratio of an expression included in the items, for each of the clusters, and

wherein the text feature vector has the extracted expression as an attribute and the acquired appearance ratio as an attribute value for each of the clusters.

5. The method according to claim 1, wherein the date expression extracted from the Web pages of each of the clusters coincides with any one of a plurality of date expression forms which are predetermined.

6. The method according to claim 1, wherein the date expression is replaced with a date-and-time expression formed by adding a time expression to the date expression.

7. An apparatus that generates time-series data from Web pages, comprising:

a processor;

a memory coupled to the processor storing instructions causing the processor to implement the following functionalities;

a user interface which receives a collection condition from a user;

a collection unit to collect Web pages, which match the collection condition received by the user interface, from a plurality of Web sites;

a data storage which stores a set of Web pages collected by the collection unit;

a data dividing unit to divide the set of Web pages stored in the data storage into a plurality of clusters, based on URL information of the Web pages;

a determining unit to extract a date expression from Web pages included in each of the clusters and determine a typical date expression form for each of the clusters, based on the extracted date expression;

an item dividing unit to divide the Web pages included in each of the clusters into a plurality of items with reference to a location where a date expression of the date expression form appears, based on the date expression form determined by the determining unit;

a time-series data generating unit to generate time-series data for each of the clusters by sorting the items, which are obtained by the item dividing unit, for each of the clusters in order of time, based on date expressions corresponding to the items;

a providing unit to provide the Web pages stored in the storage device with URL features, which represent features of URL information of the Web pages, as features of the Web pages, the providing unit including an extracting module to extract the URL features by dividing the URL information; and

a generating unit to generate feature vectors, which represent the URL features provided for the Web pages, as URL feature vectors, based on the URL features,

wherein;

the data dividing unit is to divide the set of Web pages corresponding to the URL feature vector of each of the Web pages into a plurality of clusters, based on the URL feature vector;

the URL features each includes part of URL information from which each of the URL features is extracted, as an attribute;

the URL feature vectors have all attributes of the URL features in common without any redundancy; and

the providing unit includes;

a dividing module to divide the URL information into a plurality of divided character strings by a plurality of delimiters; and

a determining module to determine each of the divided character strings as one of an attribute and an attribute value of each of the URL features, thereby to set each of the divided character strings to one of an attribute having presence or absence of a character string as an attribute value, an attribute having a divided character string subsequent to one of the delimiters as an attribute value, and an attribute value having a divided character string precedent to one of the delimiters as an attribute, in accordance with types of the delimiters, and wherein;

the delimiters are classified into two types of a first delimiter and a second delimiter;

the dividing module is to divide the URL information into a plurality of divided character strings by the first delimiter, thereby to divide some of the divided character strings, which include the second delimiter, into a pair of divided character strings by the second delimiter; and

the determining module is to determine the divided character strings, which are obtained by the first delimiter and exclude the second delimiter, as an attribute, thereby to determine presence or absence of the character strings as an attribute value of the attribute, the determining module being to determine one of the divided character strings obtained by the second delimiter, which is precedent to the second delimiter, as an attribute, thereby to determine other of the divided character strings, which is subsequent to the second delimiter, as an attribute value.

8. The apparatus according to claim 7, wherein the data dividing unit computes a degree of similarity between the clusters, based on text information of Web pages included in the clusters, and repeats a process of integrating clusters whose degree of similarity exceeds a given threshold value into a new cluster until clusters whose degree of similarity exceeds the given threshold value disappear.

9. The apparatus according to claim 7, further comprising a date expression storage which stores in advance date expression form information representing a plurality of date expression forms, and

wherein the determining unit extracts a date expression which coincides with any one of the date expression forms which are represented by the date expression form information stored in the date expression storage, from the Web pages included in each of the clusters.

10. The apparatus according to claim 7, wherein the date expression is replaced with a date-and-time expression formed by adding a time expression to the date expression.