Click or skip evaluation of synonym rules

• US 8,965,882 B1
• Filed: 11/22/2011
• Issued: 02/24/2015
• Est. Priority Date: 07/13/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Popovici, Dan, Gottweis, Juraj
• Primary Examiner(s): Brown, Sheree
• Application Number:
  US13/302,935
• Time in Patent Office: 1,190 Days
• Field of Search: 707/723, 707/769, 707/765, 707/768
• US Class Current: 707/723
• CPC Class Codes: G06F 16/3325 Reformulation based on results of preceding query
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/374 Thesaurus
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for click or skip evaluation of synonyms. In one aspect, a method includes accessing query log data that identifies, for an initial search query, a revised search query that includes one or more synonyms, and search results provided to a user. An indication that the user has selected a particular search result is received. A determination is made that the particular search result includes a particular synonym for a query term that occurs in the revised search query, and not (i) the query term or (ii) any other synonym for the query term that occurs in the revised search query. A crucial click count is assigned to a synonym rule that corresponds to the particular synonym.

Claims

1. A computer-implemented method comprising:

accessing, by one or more computers, query log data that identifies, for an initial search query;

a query term included in the initial search query,

one or more synonyms for the query term that were generated using a corresponding one or more synonym rules,

a revised search query that includes the one or more synonyms for the query term,

search results that were generated using the initial search query and the revised search query, and

a particular search result that was selected by a user;

determining, by the one or more computers, that the particular search result includes a particular synonym of the one or more synonyms for the query term that are included in the revised search query;

determining, by the one or more computers, that the particular search result includes neither (i) the query term included in the initial search query nor (ii) any of the one or more synonyms for the query term other than the particular synonym;

in response to determining that the particular search result includes the particular synonym of the one or more synonyms for the query term that are included in the revised search query, and in response to determining that the particular search result includes neither (i) the query term included in the initial search query nor (ii) any of the one or more synonyms for the query term other than the particular synonym, adjusting, by the one or more computers, a crucial click count for a synonym rule that corresponds to the particular synonym; and

determining, by the one or more computers and based at least on the crucial click count for the synonym rule that corresponds to the particular synonym, whether to revise a search query using the synonym rule that corresponds to the particular synonym.

2. The method of claim 1, comprising assigning a score to the synonym rule based at least on the crucial click count for the synonym rule.

3. The method of claim 2, wherein the score assigned to the synonym rule is based on a ratio of (i) the crucial click count to (ii) the crucial click count and a crucial skip count.

4. The method of claim 2, comprising:

determining that the score assigned to the synonym rule does not meet a threshold; and

removing the synonym rule from a set of synonym rules used to generate revised search queries based on determining that the score assigned to the synonym rule does not meet the threshold.

5. The method of claim 1, comprising:

determining that the particular search result includes a second particular synonym of the one or more synonyms for the query term that are included in the revised search query, and determining that the particular search result does not include the query term included in the initial search query; and

in response to determining that the particular search result includes the second particular synonym and does not include the query term included in the initial search query, adjusting a click count for the synonym rule that corresponds to the particular synonym.

6. The method of claim 5, comprising:

associating a first weight with the click count and a second different weight with the crucial click count; and

assigning a score to the synonym rule based at least on the crucial click count and associated first weight and the click count and associated second weight.

7. The method of claim 6, wherein the score satisfies:

8. The method of claim 1, comprising:

determining that the particular search result includes a second particular synonym of one or more synonyms for a second query term included in the initial search query that are included in the revised search query, wherein the one or more synonyms for the second query term were generated using a corresponding one or more synonym rules;

determining that the particular search result includes neither (i) the second query term included in the initial search query nor (ii) any of the one or more synonyms of the second query term other than the second particular synonym;

in response to determining that the particular search result includes the second particular synonym of the one or more synonyms for the second query term that are included in the revised search query, and in response to determining that the particular search result includes neither (i) the second query term included in the initial search query nor (ii) any of the one or more synonyms for the second query term other than the second particular synonym, adjusting a crucial click count for a second synonym rule that corresponds to the second particular synonym; and

determining, based at least on the crucial click count for the second synonym rule that corresponds to the second particular synonym, whether to revise a search query using the second synonym rule that corresponds to the second particular synonym.

9. The method of claim 1, comprising:

determining that the particular synonym was generated for the query term using a synonym rule that is associated with a particular query context, wherein adjusting the crucial click count for the synonym rule comprises adjusting a crucial click count for the synonym rule that is associated with the particular query context.

10. The method of claim 1, comprising:

determining that the particular synonym was generated for the query term using a second synonym rule that is associated with a second particular query context; and

adjusting a crucial click count for the second synonym rule that is associated with the second particular query context.

27. The method of claim 6, wherein the score satisfies:

41. The method of claim 1, wherein adjusting the crucial click count for the synonym rule that corresponding to the particular synonym comprises:

adjusting a crucial count corresponding to a synonym rule that indicates that the particular synonym is a synonym for the query term.

11. A computer-implemented method comprising:

accessing, by one or more computers, query log data that identifies, for an initial search query;

a query term included in the initial search query,

one or more synonyms for the query term that were generated using a corresponding one or more synonym rules,

a revised search query that includes the one or more synonyms for the query term,

search results that were generated using the initial search query and the revised search query,

a ranking of the search results, and

a particular search result that was selected by a user;

determining, by the one or more computers, that a search result that is ranked above the particular search result includes a particular synonym of the one or more synonyms for the query term that are included in the revised search query;

determining, by the one or more computers, that the search result that is ranked above the particular search result includes neither (i) the query term included in the initial search query nor (ii) any of the one or more synonyms for the query term other than the particular synonym;

in response to determining that the search result that is ranked above the particular search result includes the particular synonym of the one or more synonyms for the query term that are included in the revised search query, and in response to determining that the search result that is ranked above the particular search result includes neither (i) the query term included in the initial search query nor (ii) any of the one or more synonyms for the query term other than the particular synonym, adjusting, by the one or more computers, a crucial skip count for the synonym rule that corresponds to the particular synonym; and

determining, by the one or more computers and based at least on the crucial skip count for the synonym rule that corresponds to the particular synonym, whether to revise a search query using the synonym rule that corresponds to the particular synonym.

12. The method of claim 11, comprising assigning a score to the synonym rule based at least on the crucial skip count for the synonym rule.

13. The method of claim 12, wherein the score assigned to the synonym rule is based on a ratio of (i) the crucial skip count to (ii) the crucial skip count and a crucial click count.

14. The method of claim 12, comprising:

determining that the score assigned to the synonym rule does not meet a threshold; and

removing the synonym rule from a set of synonym rules used to generate revised search queries based on determining that the score assigned to the synonym rule does not meet the threshold.

15. The method of claim 11, comprising:

determining that the search result that is ranked above the particular search result includes a second particular synonym of the one or more synonyms for the query term that are included in the revised search query, and determining that the search result that is ranked above the particular search result does not include the query term included in the initial search query; and

in response to determining that the search result that is ranked above the particular search result includes the second particular synonym and does not include the query term included in the initial search query, adjusting a skip count for the synonym rule that corresponds to the particular synonym.

16. The method of claim 15, comprising:

associating a first weight with the skip count and a second different weight with the crucial skip count; and

assigning a score to the synonym rule based at least on the crucial skip count and associated first weight and the skip count and associated second weight.

17. The method of claim 16, wherein the score satisfies:

18. The method of claim 11, comprising:

determining that the search result that is ranked above the particular search result includes a second particular synonym of one or more synonyms for a second query term included in the initial search query that are included in the revised search query, wherein the one or more synonyms for the second query term were generated using a corresponding one or more synonym rules;

determining that the search result that is ranked above the particular search result includes neither (i) the second query term included in the initial search query nor (ii) any of the one or more synonyms of the second query term other than the second particular synonym;

in response to determining that the search result that is ranked above the particular search result includes the second particular synonym of the one or more synonyms for the second query term that are included in the revised search query, and in response to determining that the search result that is ranked above the particular search result includes neither (i) the second query term included in the initial search query nor (ii) any of the one or more synonyms of the second query term other than the second particular synonym, adjusting a crucial skip count for a second synonym rule that corresponds to the second particular synonym; and

determining, based at least on the crucial skip count for the second synonym rule that corresponds to the second particular synonym, whether to revise a search query using the second synonym rule that corresponds to the second particular synonym.

19. The method of claim 11, comprising:

determining that the particular synonym was generated for the query term using a synonym rule that is associated with a particular query context, wherein adjusting the crucial skip count for the synonym rule comprises adjusting a crucial skip count for the synonym rule that is associated with the particular query context.

20. The method of claim 11, comprising:

determining that the particular synonym was generated for the query term using a second synonym rule that is associated with a second particular query context; and

adjusting a crucial skip count for the second synonym rule that is associated with the second particular query context.

37. The method of claim 16, wherein the score satisfies:

21. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

accessing, by one or more computers, query log data that identifies, for an initial search query;

a query term included in the initial search query,

one or more synonyms for the query term that were generated using a corresponding one or more synonym rules,

a revised search query that includes the one or more synonyms for the query term,

search results that were generated using the initial search query and the revised search query, and

a particular search result that was selected by a user;

determining, by the one or more computers, that the particular search result includes a particular synonym of the one or more synonyms for the query term that are included in the revised search query;

determining, by the one or more computers, that the particular search result includes neither (i) the query term included in the initial search query nor (ii) any of the one or more synonyms for the query term other than the particular synonym;

in response to determining that the particular search result includes the particular synonym of the one or more synonyms for the query term that are included in the revised search query, and in response to determining that the particular search result includes neither (i) the query term included in the initial search query nor (ii) any of the one or more synonyms for the query term other than the particular synonym, adjusting, by the one or more computers, a crucial click count for a synonym rule that corresponds to the particular synonym; and

determining, by the one or more computers and based at least on the crucial click count for the synonym rule that corresponds to the particular synonym, whether to revise a search query using the synonym rule that corresponds to the particular synonym.

22. The system of claim 21, wherein the operations comprise assigning a score to the synonym rule based at least on the crucial click count for the synonym rule.

23. The system of claim 22, wherein the score assigned to the synonym rule is based on a ratio of (i) the crucial click count to (ii) the crucial click count and a crucial skip count.

24. The system of claim 22, wherein the operations comprise:

determining that the score assigned to the synonym rule does not meet a threshold; and

removing the synonym rule from a set of synonym rules used to generate revised search queries based on determining that the score assigned to the synonym rule does not meet the threshold.

25. The system of claim 21, wherein the operations comprise:

determining that the particular search result includes a second particular synonym of the one or more synonyms for the query term that are included in the revised search query, and determining that the particular search result does not include the query term included in the initial search query; and

in response to determining that the particular search result includes the second particular synonym and does not include the query term included in the initial search query, adjusting a click count for the synonym rule that corresponds to the particular synonym.

26. The system of claim 25, wherein the operations comprise:

associating a first weight with the click count and a second different weight with the crucial click count; and

assigning a score to the synonym rule based at least on the crucial click count and associated first weight and the click count and associated second weight.

28. The system of claim 21, wherein the operations comprise:

determining that the particular search result includes a second particular synonym of one or more synonyms for a second query term included in the initial search query that are included in the revised search query, wherein the one or more synonyms for the second query term were generated using a corresponding one or more synonym rules;

determining that the particular search result includes neither (i) the second query term included in the initial search query nor (ii) any of the one or more synonyms of the second query term other than the second particular synonym;

in response to determining that the particular search result includes the second particular synonym of the one or more synonyms for the second query term that are included in the revised search query, and in response to determining that the particular search result includes neither (i) the second query term included in the initial search query nor (ii) any of the one or more synonyms for the second query term other than the second particular synonym, adjusting a crucial click count for a second synonym rule that corresponds to the second particular synonym; and

determining, based at least on the adjusted click count for the second synonym rule that corresponds to the second particular synonym, whether to revise a search query using the second synonym rule that corresponds to the second particular synonym.

29. The system of claim 21, wherein the operations comprise:

determining that the particular synonym was generated for the query term using a synonym rule that is associated with a particular query context, wherein adjusting the crucial click count for the synonym rule comprises adjusting a crucial click count for the synonym rule that is associated with the particular query context.

30. The system of claim 21, wherein the operations comprise:

determining that the particular synonym was generated for the query term using a second synonym rule that is associated with a second particular query context; and

adjusting a crucial click count for the second synonym rule that is associated with the second particular query context.

31. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

accessing, by one or more computers, query log data that identifies, for an initial search query;

a query term included in the initial search query,

one or more synonyms for the query term that were generated using a corresponding one or more synonym rules,

a revised search query that includes the one or more synonyms for the query term,

search results that were generated using the initial search query and the revised search query,

a ranking of the search results, and

a particular search result that was selected by a user;

determining, by the one or more computers, that a search result that is ranked above the particular search result includes a particular synonym of the one or more synonyms for the query term that are included in the revised search query;

determining, by the one or more computers, that the search result that is ranked above the particular search result includes neither (i) the query term included in the initial search query nor (ii) any of the one or more synonyms for the query term other than the particular synonym;

in response to determining that the search result that is ranked above the particular search result includes the particular synonym of the one or more synonyms for the query term that are included in the revised search query, and in response to determining that the search result that is ranked above the particular search result includes neither (i) the query term included in the initial search query nor (ii) any of the one or more synonyms for the query term other than the particular synonym, adjusting, by the one or more computers, a crucial skip count for the synonym rule that corresponds to the particular synonym; and

determining, by the one or more computers and based at least on the crucial skip count for the synonym rule that corresponds to the particular synonym, whether to revise a search query using the synonym rule that corresponds to the particular synonym.

32. The system of claim 31, wherein the operations comprise assigning a score to the synonym rule based at least on the crucial skip count for the synonym rule.

33. The system of claim 32, wherein the score assigned to the synonym rule is based on a ratio of (i) the crucial skip count to (ii) the crucial skip count and a crucial click count.

34. The system of claim 32, wherein the operations comprise:

determining that the score assigned to the synonym rule does not meet a threshold; and

removing the synonym rule from a set of synonym rules used to generate revised search queries based on determining that the score assigned to the synonym rule does not meet the threshold.

35. The system of claim 31, wherein the operations comprise:

determining that the search result that is ranked above the particular search result includes a second particular synonym of the one or more synonyms for the query term that are included in the revised search query, and determining that the search result that is ranked above the particular search result does not include the query term included in the initial search query; and

in response to determining that the search result that is ranked above the particular search result includes the second particular synonym and does not include the query term included in the initial search query, adjusting a skip count for the synonym rule that corresponds to the particular synonym.

36. The system of claim 35, wherein the operations comprise:

associating a first weight with the skip count and a second different weight with the crucial skip count; and

assigning a score to the synonym rule based at least on the crucial skip count and associated first weight and the skip count and associated second weight.

38. The system of claim 31, wherein the operations comprise:

determining that the search result that is ranked above the particular search result includes a second particular synonym of one or more synonyms for a second query term included in the initial search query that are included in the revised search query, wherein the one or more synonyms for the second query term were generated using a corresponding one or more synonym rules;

determining that the search result that is ranked above the particular search result includes neither (i) the second query term included in the initial search query nor (ii) any of the one or more synonyms of the second query term other than the second particular synonym;

in response to determining that the search result that is ranked above the particular search result includes the second particular synonym of the one or more synonyms for the second query term that are included in the revised search query, and in response to determining that the search result that is ranked above the particular search result includes neither (i) the second query term included in the initial search query nor (ii) any of the one or more synonyms of the second query term other than the second particular synonym, adjusting a crucial skip count for a second synonym rule that corresponds to the second particular synonym; and

determining, based at least on the crucial skip count for the second synonym rule that corresponds to the second particular synonym, whether to revise a search query using the second synonym rule that corresponds to the second particular synonym.

39. The system of claim 31, wherein the operations comprise:

determining that the particular synonym was generated for the query term using a synonym rule that is associated with a particular query context, wherein adjusting the crucial skip count for the synonym rule comprises adjusting a crucial skip count for the synonym rule that is associated with the particular query context.

40. The system of claim 31, wherein the operations comprise:

determining that the particular synonym was generated for the query term using a second synonym rule that is associated with a second particular query context; and

adjusting a crucial skip count for the second synonym rule that is associated with the second particular query context.