Systems and methods for enhancing web-based searching

• US 7,613,687 B2
• Filed: 10/26/2004
• Issued: 11/03/2009
• Est. Priority Date: 05/30/2003
• Status: Expired due to Fees

• Current Assignee: Truelocal Incorporated
• Original Assignee: Truelocal Incorporated
• Inventors: Nye, Timothy G.
• Primary Examiner(s): Fleurantin; Jean B.
• Assistant Examiner(s): Kerzhner; Aleksandr
• Application Number:
  US10/973,660
• Publication Number:
  US 20050120006A1
• Time in Patent Office: 1,834 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A system for enhancing web-based searching is provided. Categorizing and clustering techniques are used to optimize searching. Businesses are classified using a control group of predetermined categories. The predetermined categories may be SIC codes or headings that are used to describe business activities. The website addresses for a business listed in the control group is determined, and the content of the business'"'"'s website is extracted. The extracted content is associated with the predetermined category that the business is classified under. The extracted content is used to further enhance the overall classification scheme. The system may compare and match the extracted content with content of other business'"'"' websites, which are similarly categorized. If a relevant keyword match is identified in several of the websites, the keyword may be used to update the classification scheme. A new category or sub-category can be created based on this keyword. Furthermore, when a search is performed, the search results are organized by these categories, and using various processes, the most common results are kept and the less relevant results are discarded.

Claims

1. An information gathering system implemented in a computer system for optimizing searching comprising:

a processor and memory;

a data extraction tool executing in the computer system, in communication with a database, extracting website content to enable full text searching, the website content being extracted from a plurality of websites associated with business entities that are classified according to a standard industry classification system (SIC), which is a predefined taxonomy of business activities having verified information about the business entities;

the database, in communication with the data extraction tool, storing the extracted website content according to a classification system that is based on the predefined taxonomy of SIC business activities;

a content analyzer, in communication with the database, identifying commonly occurring keywords in the extracted website content from the websites of business entities that are similarly classified in the SIC predefined taxonomy of SIC business activities, where the commonly occurring keywords identified are used to update the classification system, the updated classification system being used to optimize searching in response to search queries;

the content analyzer identifying commonly occurring keywords that are used to create a new category to update the classification system by;

identifying keyword matches in the extracted website content by identifying any commonly occurring keywords or phrases in the extracted website content; and

processing the matches identified by determining whether any of the keywords or phrases in the identified matches contain one or more keywords associated with any of the business activities in the SIC predefined taxonomy; and

a full text indexed search engine, in communication with the database, processing a search query by matching the search query against the database, where at least a portion of the search results are clustered based on their respective SIC business activity category.

2. An information gathering system for optimizing searching as in claim 1 wherein the SIC predefined taxonomy of SIC business activities are not computer generated.

3. An information gathering system for optimizing searching as in claim 1 wherein storing data concerning business entities according to a classification system further includes storing verified attributes about the business entities including at least one of:

business name, business address, business telephone number, or business website address.

4. An information gathering system for optimizing searching as in claim 1 wherein the business entities correspond to corporations, agencies, or organizations.

5. An information gathering system for optimizing searching as in claim 1 wherein each of the business activities in the SIC predefined taxonomy corresponds to a business descriptor.

6. An information gathering system for optimizing searching as in claim 1 further includes a search handler which identifies the website content and causes the data extraction tool to extract the identified website content.

7. An information gathering system for optimizing searching as in claim 6 wherein the search handler identifies the website content through a search engine.

8. An information gathering system for optimizing searching as in claim 7 wherein the data extraction tool extracts the website content from indexed data provided by the search engine.

9. An information gathering system for optimizing searching as in claim 6 wherein the search handler identifies the website content through a website address of the business entity.

10. An information gathering system for optimizing searching as in claim 1 wherein processing matches identified further includes a process for eliminating irrelevant matches.

11. An information gathering system for optimizing searching as in claim 1 wherein matches identified further includes a process for:

ranking each of the matches identified; and

processing the ranked matches to determine the highest ranking match.

12. An information gathering system for optimizing searching as in claim 1 wherein processing matches identified further includes the following processes:

determining whether any of the identified matches were extracted from two or more websites that are associated with any of the same business activities in the SIC predefined taxonomy;

computing the identified matches and determining whether any of the matches exceed a certain percentage of the computed total;

or

computing the identified matches and eliminating matches that are below a certain percentage of the computed total.

13. An information gathering system for optimizing searching as in claim 11 wherein updating the classification system further includes using the highest ranking match to update the classification system by creating a new category in the classification system based on the highest ranking match.

14. An information gathering system for optimizing searching as in claim 13 wherein the update to the classification system causes a sub-category of one of the business activities in the SIC predefined taxonomy to be created, where the sub-category is indicative of an association between one of the business activities of the SIC predefined taxonomy and the extracted content.

15. An information gathering system for optimizing searching as in claim 1 further including a search process that locates websites of business entities that have not been categorized in the SIC predefined taxonomy of SIC business activities.

16. An information gathering system for optimizing searching as in claim 15 wherein the data extraction tool extracts content from the websites that have not been categorized in the SIC predefined taxonomy of SIC business activities.

17. An information gathering system for optimizing searching as in claim 16 wherein the extracted content from the websites that have not been categorized in the SIC predefined taxonomy of SIC business activities is used to update the classification system.

18. An information gathering system for optimizing searching as in claim 1 further includes a search process to identify business entities associated with one or more of the business activities in the SIC predefined taxonomy, which do not have a website.

19. An information gathering system for optimizing searching as in claim 18 wherein the process to identify business entities associated with one or more of the business activities in the SIC predefined taxonomy, which do not have a website, further includes logic to assign each of the business entities identified to new categories created in the updated classification system.

20. An information gathering system for optimizing searching as in claim 1 wherein the classification system includes categories based on geographical locations, where the geographical locations are used to eliminate false positives in search results resulting from a keyword search query when one of the terms in the keyword search query has more than one meaning in the classification system.

21. An information gathering system for optimizing searching as in claim 1 wherein the classification system further includes logic that processes queries to identify business entities that are geographically located within a certain distance from a particular location.

22. An information gathering system for optimizing searching as in claim 1 wherein the updated classification system is used by the full text indexed search engine to process queries for business entities.

23. An information gathering system for optimizing searching as in claim 22 wherein the query includes attributes matching one or more categories in the updated classification system, where one or more business entities is responsive to the query.

24. An information gathering system for optimizing searching as in claim 1 wherein the categories in the updated classification are used to cluster search result hits.

25. An information gathering system for optimizing searching as in claim 1 wherein the full text indexed search engine further includes:

a search handler being responsive to a user search query; and

a clustering engine, in communication with the search handler, processing search result hits resulting from the query, where the clustering engine clusters the search result hits based on categories defined in the classification system.

26. An information gathering system for optimizing searching as in claim 1 wherein the full text indexed search engine processing a search query by matching the search query against the database, where at least a portion of the search results are clustered further includes:

returning a set of indexed results having content matching the search query;

matching the returned indexed results to the SIC codes in the database to determine hits; and

grouping the hits according to their respective SIC code.

27. An information gathering system for optimizing searching as in claim 26 wherein grouping the hits according to their respective SIC code further includes:

determining a group hit count for each group;

determining if the group hit count is above a threshold value;

displaying groups of hits that are above the threshold value, where each of the displayed groups of hits are displayed with a heading corresponding to their respective SIC code and with counts showing the number of hits in the group, where the SIC code heading displayed is an associated SIC business descriptor.

28. An information gathering system for optimizing searching as in claim 27 further includes responding to a selection of one of the SIC code headings by presenting hits in the group corresponding to the selected SIC code heading.

29. An information gathering system for optimizing searching as in claim 27 wherein grouping the hits according to their respective SIC code further includes using the first three digits of the SIC codes to group hits.

30. An information gathering system for optimizing searching as in claim 29 wherein if the search fails to return one or more groups of hits having a count above the threshold value, using the first two digits of the SIC codes to group hits.

31. A method of optimizing searching in a data processing system comprising the steps of:

identifying websites of business entities that are associated with a standard industry classification system (SIC) predefined taxonomy of business activities having verified information of the business entities;

extracting website content from the websites of business entities that are associated with the SIC predefined taxonomy of SIC business activities to enable full text searching;

processing the extracted content to identify commonly occurring keywords in the extracted content from websites of business entities that have been similarly classified in the SIC predefined taxonomy of SIC business activities, where one or more of the commonly occurring keywords are used to update a classification system, where the classification system is based on the SIC predefined taxonomy of SIC business activities;

creating a new category to update the classification system by;

identifying keyword matches in the extracted website content by identifying any commonly occurring keywords or phrases in the extracted website content; and

processing the matches identified by determining whether any of the keywords or phrases in the identified matches contain one or more keywords associated with any of the business activities in the SIC predefined taxonomy;

processing a search query using full text index searching by matching the search query against at least a portion of the extracted content, where at least a portion of the search results are clustered according to their respective SIC business activity category.

32. A method of optimizing searching as in claim 31 wherein the SIC predefined taxonomy of SIC business activities are not computer generated.

33. A method of optimizing searching as in claim 31 wherein processing matches identified further includes:

determining whether any of the identified matches were extracted from two or more websites that are associated with any of the same SIC business activities in the SIC predefined taxonomy;

computing the identified matches and determining whether any of the matches exceed a certain percentage of the computed total;

or

computing the identified matches and eliminating matches that are below a certain percentage of the computed total.

34. A method of optimizing searching as in claim 33 wherein updating the classification system further includes using the highest ranking match to update the classification system by creating a new classification category based on the highest ranking match.

35. A method of optimizing searching as in claim 31 wherein the update to the classification system causes a sub-category of one of the SIC business activities in the SIC predefined taxonomy to be created, where the sub-category is indicative of an association between one of the SIC business activities in the SIC predefined taxonomy, and the extracted content.

36. A method of optimizing searching as in claim 31 further includes identifying business entities associated with one or more of the SIC business activities in the SIC predefined taxonomy, which do not have a website.

37. A method of optimizing searching as in claim 36 wherein identifying business entities associated with one or more of the SIC business activities in the SIC predefined taxonomy, which do not have a website, further assigning each of the business entities identified to new categories created in the updated classification system.

38. A method of optimizing searching as in claim 31 wherein the classification system includes categories based on geographical locations, where the geographical locations are used to eliminate false positives in search results resulting from a keyword search query if one of the terms in the keyword search query has multiple meanings in the classification system.

39. A method of optimizing searching as in claim 31 wherein the classification system further includes processing queries to identify business entities that are geographically located within a certain distance from a particular location.

40. A method of optimizing searching as in claim 31 further including:

responding to a user search query; and

processing search result hits resulting from the query by clustering the search result hits based on categories defined in the classification system.

41. A method of optimizing searching as in claim 40 wherein processing search result hits resulting from the query by clustering the search result hits based on categories defined in the classification system further includes assigning a search result hit to a category in the classification system.

42. A computer implemented system for optimizing searching comprising:

a processor and memory;

means for identifying websites of business entities that are associated with a standard industry classification system (SIC) predefined taxonomy of business activities having verified information of the business entities;

means for extracting website content from the websites of business entities associated with the SIC predefined taxonomy of SIC business activities to enable full text searching;

means for processing the extracted content to identify commonly occurring words in the extracted content from websites of business entities that have been similarly classified by the SIC predefined taxonomy of SIC business activities, where one or more of the commonly occurring keywords are used to update the classification system, where the classification scheme is based on the SIC predefined taxonomy of SIC business activities;

means for creating a new category to update the classification system by;

means for identifying keyword matches in the extracted website content by identifying any commonly occurring keywords or phrases in the extracted website content; and

means for processing the matches identified by determining whether any of the keywords or phrases in the identified matches contain one or more keywords associated with any of the business activities in the SIC predefined taxonomy;

means for processing a search query using a full text index searching by matching the search query against at least a portion of the extracted content, where at least a portion of the search results are clustered according to their respective SIC business activity category.

43. A method of optimizing searching in a data processing system comprising the steps of:

obtaining website content of business entities that are included in a control group, where the control group defines a classification system that is based on a standard industry classification system (SIC) predefined taxonomy of SIC business activities having verified information about the business entities, the classification system being used to classify and store the website content of the business entities;

searching for unclassified business entities using at least a portion of the website content of business entities in the control group;

classifying the unclassified business entities based on the portion of website content of business entities in the control group;

identifying commonly occurring keywords in website content of business entities that are similarly classified in the classification system of the control group;

allowing the control group to grow if a substantial amount of commonly occurring keywords have been identified; and

creating a new business entities category to update the classification system of the control group allowing the control group to grow by;

identifying keyword matches in the stored website content by identifying any commonly occurring keywords or phrases in the website content; and

processing the matches identified by determining whether any of the keywords or phrases in the identified matches contain one or more keywords associated with any of the SIC business activities in the SIC predefined taxonomy; and

using the control group to optimize full text index searching of the website content by organizing search results according to categories in the classification system of the control group.

44. A method of optimizing searching as in claim 43 wherein the business entities correspond to corporations, agencies, or organizations.

45. A method of optimizing searching as in claim 43 wherein the control group includes verified attributes about the business entities including at least one of:

business name, business address, business telephone number, or business website address.

46. A method of optimizing searching as in claim 43 wherein obtaining information including website content of the business entities that are included in a control group further includes extracting website content from websites of business entities in the control group.

47. A method of optimizing searching as in claim 46 wherein the SIC control group by creating new categories for the classification system based on the commonly occurring keywords that have been identified, where the step of identifying the commonly occurring keywords includes:

identifying keyword matches in the extracted website content by identifying any commonly occurring keywords or phrases in the extracted website content; and

processing the matches identified.

48. A method of optimizing searching as in claim 43 wherein classifying the website based on the information about the business entities in the control group further includes:

extracting content from a website of an unclassified business entity; and

comparing the unclassified website content with website content of the classified business entities in the control group to determine the classification of the unclassified business entity.

49. A method of optimizing searching as in claim 48 further includes defining relationships in the classification system according to geographical locations, where geographical locations are used to eliminate false positives in search results from a keyword search query if one of the terms in the keyword search query has multiple meanings in the classification system.

50. A method of optimizing searching as in claim 48 further includes using the control group to cluster search result hits in response to a user search query.

51. A method of optimizing searching in a data processing system comprising the steps of:

storing website content associated with business entities that are included in a control group, where the control group defines a classification system that is based on a standard industry classification system (SIC) predefined taxonomy of business activities having verified information about the business entities, the classification system being used to classify and store the website content of the business entities;

responding to a search request by using full text index searching of the website content;

identifying commonly occurring keywords in website content from the search results and website content of business entities in the control group that are classified in the SIC predefined taxonomy of SIC business activities;

grouping the search results based on the commonly occurring keywords identified;

using website content from the search results to allow the control group to grow if a substantial amount of groups of commonly occurring keywords have been identified; and

creating a new business entities category to update the classification system of the control group, allowing the control group to grow by;

identifying keyword matches in the stored website content by identifying any commonly occurring keywords or phrases in the website content; and

processing the matches identified by determining whether any of the keywords or phrases in the identified matches contain one or more keywords associated with any of the SIC business activities in the SIC predefined taxonomy.

52. A method of optimizing searching as in claim 51 wherein grouping the search results based on the commonly occurring keywords identified further includes:

identifying keyword or phrase matches in website content associated with the search results and website content associated with the business entities in the control group;

ranking the matches associated with each category of business entities in the control group.

53. A method of optimizing searching as in claim 52 using website content from the search results to allow the control group to grow if a substantial amount of commonly occurring keywords have been identified further includes:

determining a SIC business activities category in the SIC predefined taxonomy having a highest ranking match of commonly occurring keywords; and

enabling the category having the highest ranking match to grow if a large group of search results are assigned to the category.