Automated processing of appropriateness determination of content for search listings in wide area network searches

• US 6,983,280 B2
• Filed: 09/13/2002
• Issued: 01/03/2006
• Est. Priority Date: 09/13/2002
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Overture Services Incorporated (Apollo Global Management, Inc.)
• Inventors: Cunningham, Stephan, Cheung, Dominic, Zhang, Jie, Wu, Jennifer, Wu, Dennis, Lang, Alan, Goodwine, Peter, Sublette, Carey, Laffoon, Barry, Holmes, Bruce T., Snell, Scott, Wang, Pierre, Wong, Wai-Yin
• Primary Examiner(s): Robinson, Greta
• Assistant Examiner(s): Lewis, Cheryl
• Application Number:
  US10/244,051
• Publication Number:
  US 20040054661A1
• Time in Patent Office: 1,208 Days
• Field of Search: 707/1, 707 3- 5, 707/10, 707/6, 709/303, 704/4, 704/6, 704/9, 715/501.1, 715/513
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A method and system for improving the efficiency of a database processing system for evaluating candidate data items representing search listings that are submitted for inclusion into a search engine database. Candidate search listings are automatically assessed for quality, style, and relevance to evaluate risk of unfavorable reception by a user and of potential exposure volume. Search listings which are higher-risk or higher-volume are routed through manual editorial review while lower-risk, lower-volume search listings are routed for immediate inclusion in the search database without manual editorial evaluation. Accordingly, human editorial efforts can be devoted to manual review of high-risk or high-volume search listings while efficiency is simultaneously improved in the processing system as a whole.

Claims

1. A computer implemented method for evaluating fitness of a data item for inclusion in a network-accessible database, the method comprising:

determining a likelihood that the data item comports with a predetermined content policy;

comparing the likelihood to a predetermined threshold; and

upon a condition in which the likelihood is below the predetermined threshold,

including the data item in the network-accessible database without human review of the data item.

2. The method of claim 1 wherein the data item is a search listing and the network-accessible database includes a computerized network search engine database.

3. The method of claim 1 wherein the network-accessible database is accessible through a hypertext transport protocol.

4. The method of claim 1 wherein the network-accessible database is accessible through the Internet.

5. The method of claim 1 wherein the data item includes textual content.

6. The method of claim 1 wherein the data item is a computerized document.

7. The method of claim 1 wherein the data item is a computerized document comporting with a hypertext markup language.

8. The method of claim 1 further comprising:

upon a condition in which the likelihood is at least the predetermined threshold,

requiring human review of the data item to determine whether the data item comports with the predetermined content policy.

9. The method of claim 8 wherein the predetermined content policy prohibits one or more blocked terms;

and

further wherein the likelihood is at least the predetermined threshold upon a determination that the data item includes data representing at least one of the blocked terms.

10. The method of claim 9 wherein the data item is further determined to be unfit for inclusion in the network-accessible database upon a determination that the data item includes data representing at least one of the blocked terms.

11. The method of claim 8 wherein the predetermined content policy requires human review of suspect content which is identified by one or more suspect terms;

and

further wherein the likelihood is at least the predetermined threshold upon a determination that the data item includes data representing at least one of the suspect terms, thereby requiring human review of the data item.

12. The method of claim 8 wherein determining comprises determining whether the data item includes data representing one or more predetermined sexual terms;

and

further wherein the likelihood is at least the predetermined threshold upon determination that the data item includes data representing at least one of the predetermined sexual terms.

13. The method of claim 8 wherein determining comprises determining whether the data item includes data representing one or more predetermined gambling terms;

and

further wherein the likelihood is at least the predetermined threshold upon determination that the data item includes data representing at least one of the predetermined gambling terms.

14. The method of claim 1 further comprising:

determining that the data item includes nonsensical text; and

rejecting the data item for inclusion in the network-accessible database.

15. The method of claim 1 further comprising:

modifying the data item to comport with the predetermined content policy;

wherein determining comprises determining the likelihood that the data item as modified comports with the predetermined content policy.

16. The method of claim 1 further comprising:

predicting a frequency of access of the data item;

comparing the predicted frequency of access to a predetermined threshold access frequency;

upon a condition in which the predicted frequency is at least the predetermined threshold, requiring human review of the data item to determine whether the data item comports with the predetermined content policy regardless of the likelihood that the data item comports with the predetermined content policy.

17. A computer implemented method for evaluating fitness of a data item for inclusion in a network-accessible database, the method comprising:

determining a likelihood that the data item comports with a predetermined content policy;

comparing the likelihood to a predetermined threshold; and

upon a condition in which the likelihood is at least the predetermined threshold, requiring human review of the data item to determine whether the data item comports with the predetermined content policy.

18. The method of claim 17 wherein the predetermined content policy prohibits one or more blocked terms;

and

further wherein the likelihood is at least the predetermined threshold upon a determination that the data item includes data representing at least one of the blocked terms.

19. The method of claim 18 wherein the data item is further determined to be unfit for inclusion in the network-accessible database upon a determination that the data item includes data representing at least one of the blocked terms.

20. The method of claim 17 wherein the predetermined content policy requires human review of suspect content which is identified by one or more suspect terms;

and

further wherein the likelihood is at least the predetermined threshold upon a determination that the data item includes data representing at least one of the suspect terms, thereby requiring human review of the data item.

21. The method of claim 17 wherein determining comprises determining whether the data item includes data representing one or more predetermined sexual terms;

and

further wherein the likelihood is at least the predetermined threshold upon determination that the data item includes data representing at least one of the predetermined sexual terms.

22. The method of claim 17 wherein determining comprises determining whether the data item includes data representing one or more predetermined gambling terms;

and

further wherein the likelihood is at least the predetermined threshold upon determination that the data item includes data representing at least one of the predetermined gambling terms.

23. A computer implemented method for evaluating fitness of a data item for inclusion in a network-accessible database, the method comprising:

determining that the data item violates a predetermined content policy;

modifying the data item to comport with the predetermined content policy;

determining that the data item as modified is fit for inclusion in the network-accessible database.

24. A computer implemented method for evaluating fitness of a data item for inclusion in a network-accessible database in accordance with a predetermined content policy specifies one or more conditions under which human review of the data item is required for evaluation of fitness of the data item for inclusion in the network-accessible database, the method comprising:

determining that the data item represents one or more violations of the predetermined content policy;

routing the data item for human review for evaluation of fitness of the data item for inclusion in the network-accessible database.

25. The method of claim 24 wherein the predetermined content policy requires human review of suspect content which is identified by one or more suspect terms;

and

further wherein determining comprises determination that the data item includes data representing at least one of the suspect terms, thereby requiring human review of the data item.

26. The method of claim 24 wherein determining comprises determining whether the data item includes data representing one or more predetermined sexual terms;

and

further wherein determining comprises determination that the data item includes data representing at least one of the predetermined sexual terms.

27. The method of claim 24 wherein determining comprises determining whether the data item includes data representing one or more predetermined gambling terms;

and

further wherein determining comprises determination that the data item includes data representing at least one of the predetermined gambling terms.

28. A computer implemented method for evaluating fitness of a data item for inclusion in a network-accessible database in accordance with a predetermined content policy specifies one or more conditions under which the data item is determined to be unfit for inclusion in the network-accessible database, the method comprising:

determining that the data item represents one or more violations of the predetermined content policy;

rejecting the data item for inclusion in the network-accessible database.

29. The method of claim 28 wherein the predetermined content policy prohibits blocked content which is identified by one or more blocked terms;

and

further wherein determining comprises determination that the data item includes data representing at least one of the blocked terms and rejecting the data item for inclusion in the network-accessible database.

30. The method of claim 28 wherein the predetermined content policy prohibits nonsensical content;

and

further wherein determining comprises determination that the data item includes nonsensical content and rejecting the data item for inclusion in the network-accessible database.

31. A computer implemented method for evaluating fitness of a search listing for inclusion in a search engine database, the method comprising:

determining a likelihood that the search listing comports with a predetermined content policy;

comparing the likelihood to a predetermined threshold; and

upon a condition in which the likelihood is below the predetermined threshold, including the search listing in the network-accessible database without human review of the search listing.

32. The method of claim 31 wherein the search engine database is accessible through the Internet.

33. The method of claim 31 wherein the search listing refers to a document which includes textual content.

34. The method of claim 31 wherein the search listing refers to a computerized document.

35. The method of claim 31 wherein the search listing refers to a computerized document comporting with a hypertext markup language.

36. The method of claim 31 further comprising:

upon a condition in which the likelihood is at least the predetermined threshold,

requiring human review of the search listing to determine whether the search listing comports with the predetermined content policy.

37. The method of claim 36 wherein the predetermined content policy prohibits one or more blocked terms;

and

further wherein the likelihood is at least the predetermined threshold upon a determination that the search listing includes data representing at least one of the blocked terms.

38. The method of claim 37 wherein the search listing is further determined to be unfit for inclusion in the network-accessible database upon a determination that the search listing includes data representing at least one of the blocked terms.

39. The method of claim 36 wherein the predetermined content policy prohibits one or more blocked terms;

and

further wherein the likelihood is at least the predetermined threshold upon a determination that the search listing references a document which includes data representing at least one of the blocked terms.

40. The method of claim 39 wherein the search listing is further determined to be unfit for inclusion in the network-accessible database upon a determination that the search listing references a document which includes data representing at least one of the blocked terms.

41. The method of claim 36 wherein the predetermined content policy requires human review of suspect content which is identified by one or more suspect terms;

and

further wherein the likelihood is at least the predetermined threshold upon a determination that the search listing includes data representing at least one of the suspect terms, thereby requiring human review of the search listing.

42. The method of claim 36 wherein the predetermined content policy requires human review of suspect content which is identified by one or more suspect terms;

and

further wherein the likelihood is at least the predetermined threshold upon a determination that the search listing references a document which includes data representing at least one of the suspect terms, thereby requiring human review of the search listing.

43. The method of claim 36 wherein determining comprises determining whether the search listing includes data representing one or more predetermined sexual terms;

and

further wherein the likelihood is at least the predetermined threshold upon determination that the search listing includes data representing at least one of the predetermined sexual terms.

44. The method of claim 36 wherein determining comprises determining whether the search listing includes data representing one or more predetermined sexual terms;

and

further wherein the likelihood is at least the predetermined threshold upon determination that the search listing references a document which includes data representing at least one of the predetermined sexual terms.

45. The method of claim 36 wherein determining comprises determining whether the search listing includes data representing one or more predetermined gambling terms;

and

further wherein the likelihood is at least the predetermined threshold upon determination that the search listing includes data representing at least one of the predetermined gambling terms.

46. The method of claim 36 wherein determining comprises determining whether the search listing includes data representing one or more predetermined gambling terms;

and

further wherein the likelihood is at least the predetermined threshold upon determination that the search listing references a document which includes data representing at least one of the predetermined gambling terms.

47. The method of claim 31 further comprising:

determining that the search listing includes nonsensical text; and

rejecting the search listing for inclusion in the search engine database.

48. The method of claim 31 further comprising:

modifying the search listing to comport with the predetermined content policy;

wherein determining comprises determining the likelihood that the search listing as modified comports with the predetermined content policy.

49. The method of claim 31 further comprising:

predicting a frequency of access of the search listing;

comparing the predicted frequency of access to a predetermined threshold access frequency;

upon a condition in which the predicted frequency is at least the predetermined threshold, requiring human review of the search listing to determine whether the search listing comports with the predetermined content policy regardless of the likelihood that the search listing comports with the predetermined content policy.

50. A computer implemented method for evaluating fitness of a search listing for inclusion in a search engine database, the method comprising:

determining a likelihood that the search listing comports with a predetermined content policy;

comparing the likelihood to a predetermined threshold; and

upon a condition in which the likelihood is at least the predetermined threshold, requiring human review of the search listing to determine whether the search listing comports with the predetermined content policy.

51. The method of claim 50 wherein the predetermined content policy prohibits one or more blocked terms;

and

further wherein the likelihood is at least the predetermined threshold upon a determination that the data item includes data representing at least one of the blocked terms.

52. The method of claim 51 wherein the search listing is further determined to be unfit for inclusion in the search engine database upon a determination that the search listing includes data representing at least one of the blocked terms.

53. The method of claim 50 wherein the predetermined content policy requires human review of suspect content which is identified by one or more suspect terms;

and

further wherein the likelihood is at least the predetermined threshold upon a determination that the search listing includes data representing at least one of the suspect terms, thereby requiring human review of the search listing.

54. The method of claim 50 wherein determining comprises determining whether the data item includes data representing one or more predetermined sexual terms;

and

further wherein the likelihood is at least the predetermined threshold upon determination that the data item includes data representing at least one of the predetermined sexual terms.

55. The method of claim 54 wherein a degree of relevance of the search listing is determined using term frequency and inverse document frequency.

56. The method of claim 50 wherein determining comprises determining whether the search listing includes data representing one or more predetermined gambling terms;

and

further wherein the likelihood is at least the predetermined threshold upon determination that the search listing includes data representing at least one of the predetermined gambling terms.

57. A computer implemented method for evaluating fitness of a search listing for inclusion in a search engine database, the method comprising:

determining that the search listing violates a predetermined content policy;

modifying the search listing to comport with the predetermined content policy;

determining that the search listing as modified is fit for inclusion in the search engine database.

58. A computer implemented method for evaluating fitness of a search listing for inclusion in a search engine database in accordance with a predetermined content policy specifies one or more conditions under with human review of the search listing is required for evaluation of fitness of the search listing for inclusion in the search engine database, the method comprising:

determining that the search listing represents one or more violations of the predetermined content policy;

routing the search listing for human review for evaluation of fitness of the search listing for inclusion in the search engine database.

59. The method of claim 58 wherein the predetermined content policy requires human review of suspect content which is identified by one or more suspect terms;

and

further wherein determining comprises determination that the search listing includes data representing at least one of the suspect terms, thereby requiring human review of the search listing.

60. The method of claim 58 wherein determining comprises determining whether the search listing includes data representing one or more predetermined sexual terms;

and

further wherein determining comprises determination that the search listing includes data representing at least one of the predetermined sexual terms.

61. The method of claim 58 wherein determining comprises determining whether the search listing includes data representing one or more predetermined gambling terms;

and

further wherein determining comprises determination that the search listing includes data representing at least one of the predetermined gambling terms.

62. The method of claim 58 wherein the content policy specifies a predetermined requisite degree of relevance of the search listing.

63. The method of claim 62 wherein the search listing includes a search term and a title;

and

further wherein the search listing has a relevance which is determined at least in part by a degree of relevance between the search term and the title.

64. The method of claim 62 wherein the search listing includes a search term and a description;

and

further wherein the search listing has a relevance which is determined at least in part by a degree of relevance between the search term and the description.

65. The method of claim 62 wherein the search listing includes a search term and refers to a document;

and

further wherein the search listing has a relevance which is determined at least in part by a degree of relevance between the search term and the document.

66. The method of claim 62 wherein the search listing includes a title and a description;

and

further wherein the search listing has a relevance which is determined at least in part by a degree of relevance between the title and the description.

67. The method of claim 62 wherein the search listing includes a title and refers to a document;

and

further wherein the search listing has a relevance which is determined at least in part by a degree of relevance between the title and the document.

68. The method of claim 62 wherein the search listing includes a description and refers to a document;

and

further wherein the search listing has a relevance which is determined at least in part by a degree of relevance between the description and the document.

69. The method of claim 62 wherein the search listing includes a search term which at least partially defines a degree of relevance of the search listing, the method further comprising:

determining that the degree of relevance of the search listing is below a predetermined threshold relevance; and

adjusting the degree of relevance of the search listing using a semantic alternative of the search term.

70. The method of claim 69 wherein the semantic alternative is a synonym of the search term.

71. The method of claim 69 wherein the semantic alternative is a hyponym of the search term.

72. The method of claim 69 wherein the semantic alternative is a meronym of the search term.

73. The method of claim 62 further comprising:

preprocessing the search listing prior to determining a degree of relevance of the search listing.

74. The method of claim 73 wherein preprocessing comprises tokenization of the search listing.

75. The method of claim 73 wherein preprocessing comprises rendering the search listing case-insensitive.

76. The method of claim 73 wherein preprocessing comprises rendering the search listing verb-tense-insensitive.

77. The method of claim 73 wherein preprocessing comprises correcting spelling of content of the search listing.

78. The method of claim 73 wherein preprocessing comprises removal of stop words from the search listing.

79. A computer implemented method for evaluating fitness of a search listing for inclusion in a search listing database in accordance with a predetermined content policy specifies one or more conditions under which the search listing is determined to be unfit for inclusion in the search engine database, the method comprising:

determined that the search listing represents one or more violations of the predetermined content policy;

rejecting the search listing for inclusion in the network-accessible database.

80. The method of claim 79 wherein the predetermined content policy prohibits blocked content which is identified by one or more blocked terms;

and

further wherein determining comprises determination that the search listing includes data representing at least one of the blocked terms and rejecting the search listing for inclusion in the search engine database.

81. The method of claim 79 wherein the predetermined content policy prohibits nonsensical content;

and

further wherein determining comprises determination that the search listing includes nonsensical content and rejecting the search listing for inclusion in the search engine database.