Method and system for crawling, mapping and extracting information associated with a business using heuristic and semantic analysis

• US 20090119268A1
• Filed: 11/04/2008
• Published: 05/07/2009
• Est. Priority Date: 11/05/2007
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Bandaru, Nagaraju, Moyer, Eric D., Radhakrishna, Shrisha
• Granted Patent: US 8,166,013 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/258 Heading extraction; Automatic titling; Numbering
• CPC Class Codes: G06Q 10/0631 Resource planning, allocation, distributing or scheduling for enterprises or organisations

Abstract

A method and system for crawling multiple websites containing one or more web pages having information relevant to a particular domain of interest, such as details about local restaurants, extracting content from such websites, such as hours, location and phone number as well as reviews, review dates and other business specific information, and associating the extracted content with a specific business entity.

Claims

1. A system for crawling, mapping, extracting and associating information on a web page with a known business, the system comprising:

means for identifying the website address of a web page to be crawled;

means for downloading and storing the contents of said web page;

means for evaluating the stored content and identifying a unique identification symbol within said stored content and associated with a business referenced on said web page;

means for transforming the stored content to a normalize form;

means for identifying one or more potential businesses that may be associated with said normalized content;

means for confidently selecting one business, from among said one or more potential businesses, to be associated with said normalized content;

means for mapping said unique identification symbol with said confidently selected business;

means for extracting one or more elements from said stored contents of said web page in accordance with an extraction template formed in accordance with the structure of said web page, said elements comprising data about the business referenced on said web page;

means for associating said extracted elements about the business referenced on said web page with the confidently selected one business; and

means for publishing the results of said association of said extracted elements with said confidently selected one business.

2. The system for crawling, mapping, extracting and associating information on a web page with a known business, as claimed in claim 1, wherein said means for evaluating the stored content and identifying a unique identification symbol within said stored content and associated with a business referenced on said web page further comprises heuristic analysis means for analyzing the underlying construction of said web page and identifying one or more links to other web pages and storing said one or more links to other web pages to be used as a web page to be crawled.

3. The system for crawling, mapping, extracting and associating information on a web page with a known business, as claimed in claim 1, wherein said means for transforming the stored content to a normalized form further includes at least one of means for transforming telephone numbers and natural language processing means for analyzing said stored content and identifying at least one of street names, street numbers, street directions, house numbers, neighborhoods, city names, state names, zip codes and points of interest.

4. The system for crawling, mapping, extracting and associating information on a web page with a known business, as claimed in claim 1, wherein said means for identifying one or more potential businesses that may be associated with said normalized content comprises a database of known businesses, and wherein said means for confidently selecting one business, from among said one or more potential businesses, to be associated with said normalized content further comprises:

means for parsing said normalized content to identify one or more tokens, each of said tokens being representative of a word or number associated with a physical address and associated telephone number;

means for semantically analyzing said tokens in accordance with heuristic rules, said semantic analysis resulting in the generation of one or more possible business identifications; and

means for scoring each of said possible business identifications, the result of said scoring indicating a business with the highest score, said business with the highest score being identified as said confidently selected one business.

5. The system for crawling, mapping, extracting and associating information on a web page with a known business, as claimed in claim 1, wherein said web page is from a structured web site where all pages are formatted in a pre-defined manner.

6. The system for crawling, mapping, extracting and associating information on a web page with a known business, as claimed in claim 1, wherein said web page is from an unstructured web site.

7. The system for crawling, mapping, extracting and associating information on a web page with a known business, as claimed in claim 1, further comprising means for identifying when the contents of a web page that has previously been stored has changed, whereby said web page may be crawled again and new contents stored, each of the other elements of claim 1 repeated with the new contents.

8. A method for collecting information about a local business from multiple web pages including:

identifying one or more web pages that are likely to contain information about said local business;

crawling said one or more web pages;

collecting and storing all of the content found on said one or more web pages;

evaluating said stored content to determine the business entity said stored content refers to;

matching said business entity to said local business;

extracting all the information about said local business from said stored content; and

publishing said extracted information, said publication including an indication of the local business to which said extracted information refers.

9. An improved system for crawling websites to collect and extract information about a specific local business, said system comprising:

means for identifying a list of websites known to contain information about local businesses;

means for crawling a seed URL located within one of said websites known to contain information about local businesses;

means for means for downloading and storing all of the content found on said website at a web page identified by said seed URL;

means for analyzing said stored content to identify all other URL'"'"'s the web page identified by said seed URL points to and to determine of said other URL'"'"'s point to a web page containing an attribute of interest about said specific local business;

means for storing said identified all other URL'"'"'s whereby they can be also used as seed URL'"'"'s;

means for extracting a business identification code from said stored content, said business identification code being a unique identifier used to organize all information about a specific local business on said website; and

means for analyzing said stored content in order to associate it with a specific local business, further comprising means for extracting at least one of a business name, business address and business phone number from said stored content;

means for comparing said extracted business name, business address and business phone number with a directory of known local businesses and identifying the known local business that said stored content is associated with;

means for extracting all other information about said identified known local business from said stored content and associating said extracted information with said identified local business; and

means for publishing the results of said association.

10. The improved system for crawling websites to collect and extract information about a specific local business, as claimed in claim 9, wherein said local business is a restaurant and wherein said web pages containing an attribute of interest about a local business include web pages containing at least one of reviews, ratings, directions, and bibliographic information such as address, telephone number and hours of operation.

11. The improved system for crawling websites to collect and extract information about a specific local business, as claimed in claim 10, wherein said means for analyzing said stored content further comprises the use of heuristic rules and semantic analysis.

15. The improved system for crawling websites to collect and extract information about a specific local business, as claimed in claim 10, wherein said means for extracting all other information about said identified local business further comprises the use of a heuristically derived extraction template.

16. The improved system for crawling websites to collect and extract information about a specific local business, as claimed in claim 9 wherein said seed URL identifies a blog.

17. The improved system for crawling websites to collect and extract information about a specific local business, as claimed in claim 16 wherein said means for identifying a list of websites known to contain information about local businesses further comprises:

means for creating a list of query strings each query string comprising the name of a local business and its city of operation;

means for submitting each query string to a notification service that monitors the publication of new or altered information in blogs, said notification service returning an indication when a web page has been identified as matching the contents of the query string, said indication including the URL of said identified web page, wherein said identified web page is identified as containing information about a local business.

12. The improved system for crawling websites to collect and extract information about a specific local business, as claimed in claim 10, wherein said means for comparing said extracted business name, business address and business phone number with a directory of known local businesses further comprise the use of semantic analysis.

12-14. -14. (canceled)