System and method for selecting a sub-domain for a specified domain of the web

• US 20070266016A1
• Filed: 05/11/2006
• Published: 11/15/2007
• Est. Priority Date: 05/11/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Negi, Sumit, Zhang, Zengyan, Holmes, Scott, Mi, Hongcheng
• Granted Patent: US 7,542,970 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access

Abstract

A selection system and method. The selection method comprises receiving, by a computing system, a taxonomy of data related to a specified domain of knowledge on the web. A taxonomy tree is constructed from the taxonomy. A sub tree related to a sub-domain from specified domain is selected from the taxonomy tree. A first list comprising user expected universal resource locators (URLs) related to the sub-domain is received. A second list comprising topic expressions defining each node of the taxonomy sub-tree is generated. A query based on the second list is generated. The query is applied on an index of URLs generated from a web crawling process to generate a third list. A recall value is calculated based on the first list and the third list.

Claims

1. A selection method, comprising:

receiving, by a computing system, a taxonomy of data related to a specified domain of knowledge on the web;

storing, by said computing system, said taxonomy of data;

constructing, by a software application within said computing system, a taxonomy tree from said taxonomy;

receiving, by said computing system, a user selection for a taxonomy sub-tree from said taxonomy tree, said sub tree related to a sub-domain from said specified domain;

receiving, by said computing system, a first list comprising user expected universal resource locators (URLs) related to said sub-domain;

generating, by said software application, a second list comprising topic expressions defining each node of said taxonomy sub-tree;

generating, by said software application, a query based on said second list by applying at least one Boolean operator on said topic expressions on said second list;

applying, by said software application, said query on an index of URLs, said index generated from a web crawling process;

generating, by said query, a third list comprising actual URLs located during said query;

determining, by said software application, a first group (A) of URLs that are listed on and common to said third list and said first list

determining, by said software application, a second group (B) of URLs that are listed on only said first list;

calculating, by said software application, a recall value (R) based on a number of URLs in said first group (N_A) and a number of URLs in said second group (N_B).

2. The method of claim 1, further comprising:

randomly sampling, by said software application, said third list to generate a sampled list (D) of URLs from said third list;

sending, said sampled list (D) to a user of said computing system;

receiving, by said computing system, a user selected sub-list (C) of URLs from said sampled list (D), said user selected sublist based on a selection criteria;

calculating, by said software application, a precision value (P) based on a number of URLs on said user selected sub-list (N_C) and a number of URLs on said sampled list (N_D);

saving, on said computing system, said sub-list (C).

3. The method of claim 2, wherein P=N_C/N_D*100%.

4. The method of claim 1, wherein R=N_A/N_D*100%.

5. The method of claim 1, wherein said taxonomy comprises XML code.

6. The method of claim 1, further comprising:

receiving, by said computing system, a fourth list comprising specified topic expressions; and

adding, by said software application, said fourth list to said second list.

7. The method of claim 1, further comprising:

receiving, by said computing system, a command to delete some of said topic expressions from said second list.

8. A computing system comprising a processor coupled to a computer-readable memory unit, said memory unit comprising a software application and instructions that when executed by the processor implement a selection method, said method comprising:

receiving, by said computing system, a taxonomy of data related to a specified domain of knowledge on the web;

storing, by said computing system, said taxonomy of data;

constructing, by a software application within said computing system, a taxonomy tree from said taxonomy;

receiving, by said computing system, a user selection for a taxonomy sub-tree from said taxonomy tree, said sub tree related to a sub-domain from said specified domain;

receiving, by said computing system, a first list comprising user expected universal resource locators (URLs) related to said sub-domain;

generating, by said software application, a second list comprising topic expressions defining each node of said taxonomy sub-tree;

generating, by said software application, a query based on said second list by applying at least one Boolean operator on said topic expressions on said second list;

applying, by said software application, said query on an index of URLs, said index generated from a web crawling process;

generating, by said query, a third list comprising actual URLs located during said query;

determining, by said software application, a first group (A) of URLs that are listed on and common to said third list and said first list

determining, by said software application, a second group (B) of URLs that are listed on only said first list;

calculating, by said software application, a recall value (R) based on a number of URLs in said first group (N_A) and a number of URLs in said second group (N_B).

9. The computing system of claim 8, wherein the method further comprises:

randomly sampling, by said software application, said third list to generate a sampled list (D) of URLs from said third list;

sending, said sampled list (D) to a user of said computing system;

receiving, by said computing system, a user selected sub-list (C) of URLs from said sampled list (D), said user selected sublist based on a selection criteria;

calculating, by said software application, a precision value (P) based on a number of URLs on said user selected sub-list (N_C) and a number of URLs on said sampled list (N_D);

saving, on said computing system, said sub-list (C).

10. The computing system of claim 9, wherein P=C/D*100%.

11. The computing system of claim 8, wherein R=A/B*100%.

12. The computing system of claim 8, wherein said taxonomy comprises XML code.

13. The computing system of claim 8, wherein said method further comprises:

receiving, by said computing system, a fourth list comprising specified topic expressions; and

adding, by said software application, said fourth list to said first list.

14. The computing system of claim 8, wherein said method further comprises:

receiving, by said computing system, a command to delete some of said topic expressions from said second list.

15. A computer program product, comprising a computer usable medium including a software application and computer readable program code embodied therein, said computer readable program code comprising an algorithm adapted to implement a selection method within a computing system, said method comprising:

receiving, by said computing system, a taxonomy of data related to a specified domain of knowledge on the web;

storing, by said computing system, said taxonomy of data;

constructing, by a software application within said computing system, a taxonomy tree from said taxonomy;

receiving, by said computing system, a user selection for a taxonomy sub-tree from said taxonomy tree, said sub tree related to a sub-domain from said specified domain;

receiving, by said computing system, a first list comprising user expected universal resource locators (URLs) related to said sub-domain;

generating, by said software application, a second list comprising topic expressions defining each node of said taxonomy sub-tree;

generating, by said software application, a query based on said second list by applying at least one Boolean operator on said topic expressions on said second list;

applying, by said software application, said query on an index of URLs, said index generated from a web crawling process;

generating, by said query, a third list comprising actual URLs located during said query;

determining, by said software application, a first group (A) of URLs that are listed on and common to said third list and said first list

determining, by said software application, a second group (B) of URLs that are listed on only said first list;

calculating, by said software application, a recall value (R) based on a number of URLs in said first group (N_A) and a number of URLs in said second group (N_B).

16. The computer program product of claim 15, wherein said method further comprises:

randomly sampling, by said software application, said third list to generate a sampled list (D) of URLs from said third list;

sending, said sampled list (D) to a user of said computing system;

receiving, by said computing system, a user selected sub-list (C) of URLs from said sampled list (D), said user selected sublist based on a selection criteria;

calculating, by said software application, a precision value (P) based on a number of URLs on said user selected sub-list (N_C) and a number of URLs on said sampled list (N_D);

saving, on said computing system, said sub-list (C).

17. The computer program product of claim 16, wherein P=C/D*100%.

18. The computer program product of claim 15, wherein R=A/B*100%.

19. The computer program product of claim 15, wherein said method further comprises:

receiving, by said computing system, a fourth list comprising specified topic expressions; and

adding, by said software application, said fourth list to said first list.

20. The computer program product of claim 15, wherein said method further comprises:

receiving, by said computing system, a command to delete some of said topic expressions from said second list.