Search engine system and associated content analysis methods for locating web pages with product offerings

• US 7,395,259 B2
• Filed: 07/30/2004
• Issued: 07/01/2008
• Est. Priority Date: 12/08/1999
• Status: Expired due to Fees

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Bailey, David R., Rajaraman, Anand, Feldman, Todd J.
• Primary Examiner(s): Pardo; Thuy N
• Application Number:
  US10/909,134
• Publication Number:
  US 20050004889A1
• Time in Patent Office: 1,432 Days
• Field of Search: 707/3, 707/4, 707/10, 707/100, 707/102, 707/2, 707/7, 707/6, 707/5, 717/101, 717/102, 705/26, 705/14, 705/5, 709/201, 709/207, 709/203
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06Q 10/02 Reservations, e.g. for tickets, services or events
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0635 Processing of requisition or of purchase orders
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A search engine system assists users in locating web pages from which user-specified products can be purchased. Web pages located by a crawler program are scored, based on a set of criteria, according to likelihood of including a product offering. A query server accesses an index of the scored web pages to locate pages that are both responsive to a user'"'"'s search query and likely to include a product offering. In one embodiment, the responsive web pages are listed on a composite search results page together with products that satisfy the query.

Claims

1. A computer-implemented method of analyzing web page content, the method comprising:

retrieving a web page located by a crawler program;

programmatically analyzing content of the web page to evaluate whether the web page includes a product offering; and

generating, based at least in part on the programmatic analysis of the web page, a score that reflects a likelihood that the web page includes a product offering.

2. The method of claim 1, wherein programmatically analyzing the content of the web page comprises checking for the existence of each of a plurality of indicia of a product offering within content of the web page.

3. The method of claim 2, wherein generating the score comprises according different weights to different indicia of a product offering found within the web page.

4. The method of claim 1, wherein programmatically analyzing content of the web page comprises determining whether the web page includes a character string indicative of currency.

5. The method of claim 1, wherein programmatically analyzing content of the web page comprises searching for a character string indicative of an offering price.

6. The method of claim 1, wherein programmatically analyzing content of the web page comprises searcfiing for at least one of the following text patterns within the web page, wherein square brackets signify a currency indicator followed by a string of numbers:

“

price is []”

, “

price;

[]”

, “

list;

[]”

, “

regularly;

[]”

, “

our price is []”

, “

price including standard shipping is []”

, “

cost is []”

, “

on sale now at []”

, “

on sale now for []”

, “

[]for one”

.

7. The method of claim 1, wherein programmatically analyzing content of the web page comprises checking for the existence of at least one of the following within the web page:

an SKU number, shipping information, sales tax information, a click-to-buy option.

8. The method of claim 1, wherein the web page includes a link to a second web page, and the score is additionally based on a programmatic analysis of content of the second web page.

9. The method of claim 1, wherein the web page is one of a plurality, of web pages of a web site, and the score is additionally based on an analysis of other web pages of the web site.

10. The method of claim 1, further comprising using the score to determine whether to include the web page in an index of web pages.

11. The method of claim 1, further comprising taking the score into consideration in determining whether to include the web page in a search results set of a search query submitted by a user.

12. The method of claim 1, further comprising taking the score into consideration in prioritizing the web page among a set of search results for presentation to a user.

13. The method of claim 1, wherein the score indicates a likelihood that the web page provides an option to purchase a product online.

14. The method of claim 1, further comprising extracting, from the web page, text that falls within a predefined distance of indicia of a product offering, and incorporating the text into an index that maps keywords to web pages.

15. The method of claim 1, further comprising programmatically analyzing the web page to identify a product category to which the web page corresponds.

16. The method of claim 1, wherein retrieving the web page comprises retrieving the web page with the crawler program.

17. A computer readable medium having stored thereon a computer program which, when executed by a computer, causes the computer to perform the method of claim 1.

49. The method of claim 1, wherein the method is performed in an off-line processing mode.

50. The method of claim 1, wherein programmatically analyzing content of the web page comprises checking the web page for preprogrammed indicia of a product offering.

51. The method of claim 1, wherein the step of programmatically analyzing content of the web page is performed in response to the web crawler program locating the web page.

18. A computer-implemented method of analyzing and indexing web pages, the method comprising:

retrieving a web page located by a crawler program;

programmatically analyzing content of at least the web page to evaluate whether the web page includes a product offering; and

if a product offering is detected in the web page as a result of the programmatic analysis, storing a representation of the web page in an index used by a search engine to provide functionality for users to substantially limit their searches to web pages that include product offerings.

19. The method of claim 18, wherein programmatically analyzing the content comprises generating a score that represents a likelihood that the web page includes a product offering.

20. The method of claim 19, wherein the web page includes a link to a second web page, and the score is additionally based on an analysis of content of the second web page.

21. The method of claim 19, wherein the method further comprises comparing the score to a selected threshold to determine whether to include the representation of the web page in the index.

22. The method of claim 18, wherein programmatically analyzing the content comprises checking for the existence of each of a plurality of indicia of a product offering within content of the web page.

23. The method of claim 18, wherein programmatically analyzing the content comprises according different weights to different indicia of a product offering found within the web page.

24. The method of claim 18, wherein programmatically analyzing the content comprises determining whether the web page includes a character string indicative of currency.

25. The method of claim 18, wherein programmatically analyzing the content comprises searching for at least one of the following text patterns within the web page, wherein square brackets signify a currency indicator followed by a string of numbers:

“

price is []”

, “

price;

[]”

, “

list;

[]”

, “

regularly;

[]”

, “

our price is []”

, “

price including standard shipping is []”

, “

cost is []”

, “

on sale now at []”

, “

on sale now for []”

, “

[] for one”

.

26. The method of claim 18, wherein programmatically analyzing the content comprises checking for the existence of at least one of the following within the web page:

an SKU number, shipping information, sales tax information, a click-to-buy option.

27. The method of claim 18, wherein storing a representation of the web page within an index comprises extracting, from the web page, text that falls within a predefined distance of indicia of a product offering, and incorporating the text into the index.

28. The method of claim 18, further comprising programmatically analyzing the web page to identify a product category to which the web page corresponds.

29. A computer readable medium having stored thereon a computer program which, when executed by a computer, causes the computer to perform the method of claim 18.

30. A computer-implemented method of analyzing web pages, the method comprising:

retrieving a plurality of web pages of a web site;

programmatically analyzing the plurality of web pages to check for predefined indicia of a product offering within content of the web pages; and

generating a score for the web site, said score reflecting a result of the programmatic analysis of the plurality of web pages.

31. The method of claim 30, wherein the score represents a likelihood that web pages of the web site include product offerings.

32. The method of claim 30, further comprising using the score to determine whether to include the web site in an index of a search engine.

33. The method of claim 30, wherein generating the score comprises according different weights to different indicia of a product offering found within the web pages.

34. The method of claim 30, wherein programmatically analyzing the web pages comprises determining whether the web pages include a character string indicative of currency.

35. The method of claim 30, wherein programmatically analyzing the web pages comprises searching the web pages for a character string indicative of an offering price.

36. The method of claim 30, wherein programmatically analyzing the web pages comprises searching for at least one of the following text patterns, wherein square brackets signify a currency indicator followed by a string of numbers:

“

price is []”

, “

price;

[]”

, “

list”

, “

regularly;

[]”

, “

our price is []”

, “

price including standard shipping is []”

, “

cost is []”

, “

on sale now at []”

, “

on sale now for []”

, “

[] for one”

.

37. The method of claim 30, wherein programmatically analyzing the web pages comprises checking for the existence of at least one of the following within the web pages:

an SKU number, shipping information, sales tax information, a click-to-buy option.

38. The method of claim 30, wherein the score represents a likelihood that the web site provides an option to purchase a product online.

39. The method of claim 30, wherein the web pages are retrieved and analyzed in response to being located by a web crawler program.

40. A computer readable medium having stored thereon a computer program which, when executed by a computer, causes the computer to perform the method of claim 30.

41. A method of processing search queries, comprising:

receiving a search query specified by a user;

identifying a set of web pages that both (a) are responsive to the search query, and (b) have respective scores that satisfy a threshold, said scores being based on an automated analysis of web page content and representing likelihoods that the web pages include product offerings; and

generating a search results page that is responsive to the search query, said search results page including a listing of at least some of the web pages in the set.

42. The method of claim 41, wherein identifying the set of web pages comprises accessing a keyword index generated by crawling the World Wide Web.

43. The method of claim 41, wherein the search results page additionally lists products located within a database of a merchant.

44. The method of claim 41, wherein identifying the set of web pages comprises using a keyword index that is substantially limited to web pages having scores that satisfy the threshold.

45. The method of claim 41, further comprising providing, in association with a web page of said set, an option for the user to rate an associated merchant.

46. A computer readable medium having stored thereon a computer program which, when executed by computer, performs the method of claim 41.

47. The method of claim 41, further comprising identifying, from at least one database of product information, a set of products that are responsive to the search query, and including a listing of at least some of the identified products on the search results page.

48. The method of claim 47, wherein the listing of identified products is displayed on the search results page separately from the list of web pages.

52. A product-oriented search engine system, comprising:

a first data repository that contains product data of known sellers of products;

a second data repository that identifies web pages that have been programmatically determined, within a selected level of confidence, to include a product offering; and

a query server that is responsive to a search query specified by a user by checking the first data repository for products that are responsive to the search query, and by checking the second data repository for web pages that both are responsive to the search query and have been determined, within said confidence level, to include a product offering.

53. The search engine system of claim 52, further comprising an executable component that makes the programmatic determination, at least in part, by analyzing web page content to check for each of a plurality of indicia of a product offering.