Determining search relevance from user feedback

• US 9,177,341 B2
• Filed: 11/13/2014
• Issued: 11/03/2015
• Est. Priority Date: 08/23/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Mason, Llewellyn J., Kuznetsova, Olga I., Commons, Peter V., Wiggins, Kelly L.
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): BUI, THUY T
• Application Number:
  US14/540,243
• Publication Number:
  US 20150127491A1
• Time in Patent Office: 355 Days
• Field of Search: 707/722, 707/728, 707/731
• US Class Current: 1/1
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/3326 using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages
• CPC Class Codes: G06F 16/337 Profile generation, learning or modification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0625 Directed, with specific intent or strategy

Abstract

Disclosed are various embodiments for using relevance indications provided by a user in performing a search. Search results corresponding to a search query are provided to a user of a client computing device. The user provides a relevance indication corresponding to relevance of at least one of the search results. A subsequent search is performed which takes the relevance indication into account.

Claims

1. A non-transitory computer-readable medium embodying a program executable in a computing device, the program comprising:

code that performs a search using a search query received in a first search request from a client device to produce a first plurality of search results, the search being associated with a catalog and the first plurality of search results corresponding to items listed in the catalog;

code that provides the first plurality of search results to the client device in response to the first search request;

code that receives a plurality of relevance indications corresponding to an indication of a respective relevance of at least one of the first plurality of search results, wherein the respective relevance is associated with a relevance score based at least in part upon search behavior data associated with previous submissions of the search query;

code that determines whether individual ones of the plurality of relevance indications meet a relevance threshold, the relevance threshold set dynamically as a point at which relevance scores associated with the first plurality of search results begin to decline by a predetermined number of score units;

code that performs a subsequent search using the search query in response to receiving a second search request while excluding individual ones of the first plurality of search results that have a respective relevance score that fails to meet the relevance threshold to produce a second plurality of search results, the second plurality of search results corresponding to items listed in the catalog;

code that provides the second plurality of search results to the client device in response to the second search request; and

code that receives a sort criterion from the client device, wherein the sort criterion comprises at least one of a price of the items listed in the catalog or a popularity of the items listed in the catalog;

code that sorts the second plurality of search results according to the sort criterion; and

code that provides the sorted second plurality of search results to the client device.

2. The non-transitory computer-readable medium of claim 1, wherein the search behavior data includes a number of occurrences of reaching the at least one of the first plurality of search results via the search query.

3. The non-transitory computer-readable medium of claim 1, wherein the program further comprises code that generates a network page that comprises the first plurality of search results, wherein the code that provides the first plurality of search results to the client device sends the generated network page to the client device.

4. The non-transitory computer-readable medium of claim 1, wherein the program further comprises code that generates a network page that comprises the second plurality of search results, wherein the code that provides the second plurality of search results to the client device sends the generated network page to the client device.

5. A method, comprising:

obtaining, via at least one of one or more computing devices, a plurality of relevance indications corresponding to an indication of a respective relevance of individual ones of a first plurality of search results corresponding to a search query, wherein the respective relevance is associated with a relevance score based at least in part upon search behavior data associated with previous submissions of the search query;

determining, via at least one of the one or more computing devices, whether individual ones of the plurality of relevance indications meet a relevance threshold, where the relevance threshold is set dynamically as a point at which relevance scores associated with the first plurality of search results begin to decline by a predetermined number of score units;

performing, via at least one of the one or more computing devices, a subsequent search to produce a second plurality of search results;

excluding, via at least one of the one or more computing devices, search results that have a relevance score that fails to meet the relevance threshold from the second plurality of search results;

receiving, via at least one of the one or more computing devices, a sort criterion from a client device, wherein the sort criterion comprises at least one of a price of items listed in a product catalog or a popularity of the items listed in the product catalog;

sorting, via at least one of the one or more computing devices, the second plurality of search results according to the sort criterion; and

providing, via at least one of the one or more computing devices, the sorted second plurality of search results to the client device.

6. The method of claim 5, wherein a value for the sort criterion is provided by a client application.

7. The method of claim 5, wherein the search behavior data includes a number of occurrences of the plurality of search results being reached via the search query.

8. The method of claim 5, wherein the relevance threshold represents an absolute number of search results.

9. The method of claim 5, wherein the relevance threshold represents a proportion of the search results.

10. The method of claim 5, wherein the relevance threshold represents a number of pages associated with a presentation of the search results.

11. The method of claim 5, wherein excluding, via at least one of the one or more computing devices, search results that have a relevance score that fails to meet the relevance threshold from the second plurality of search results comprises comparing the relevance threshold to the relevance score.

12. The method of claim 5, further comprising flagging, via at least one of the one or more computing devices, search results that have a relevance score that fails to meet the relevance threshold.

13. The method of claim 5, further comprising updating, via at least one of the one or more computing devices, at least a portion of a search index which corresponds to the search query to include the plurality of relevance indications, wherein the search index is associated with a product catalog and the plurality of search results are associated with product detail pages corresponding to items listed in the product catalog.

14. The method of claim 5, further comprising generating, via at least one of the one or more computing devices, a network page comprising the sorted second plurality of search results, wherein providing the sorted second plurality of search results to the client device further comprises sending the generated network page to the client device.

15. The method of claim 5, further comprising providing the first plurality of search results to the client device.

16. A system, comprising:

at least one computing device comprising a processor and a memory; and

an application executable in the at least one computing device, the application comprising;

logic that, in response to receiving a search query in a first search request from a client device, performs a search for the search query to produce a first plurality of search results, the search being associated with a product catalog and the first plurality of search results corresponding to items listed in the product catalog;

logic that provides the first plurality of search results corresponding to the search query to the client device;

logic that receives a plurality of relevance indications corresponding to an indication of a respective relevance of at least one of the first plurality of search results, wherein the respective relevance is associated with a relevance score based at least in part upon search behavior data associated with previous submissions of the search query;

logic that determines whether individual ones of the plurality of relevance indications meet a relevance threshold, the relevance threshold set dynamically as a point at which relevance scores associated with the first plurality of search results begin to decline by a predetermined number of score units;

logic that performs a subsequent search for the search query in response to a second search request to produce a second plurality of search results;

logic that excludes search results that have a relevance score that fails to meet the relevance threshold from the second plurality of search results;

logic that receives a sort criterion from the client device, wherein the sort criterion comprises at least one of a price of items listed in the product catalog or a popularity of the items listed in the product catalog;

logic that sorts the second plurality of search results according to the sort criterion; and

logic that provides the sorted second plurality of search results to the client device.

17. The system of claim 16, wherein the search behavior data is based at least in part upon a number of occurrences of reaching the at least one of the first plurality of search results via the search query.

18. The system of claim 16, the application further comprising logic that captures an indication that at least one of the sorted second plurality of search results is relevant, and wherein the logic that excludes removes any search results from the second plurality of search results that are less relevant than the indicated at least one of the sorted second plurality of search results.

19. The system of claim 16, wherein the application further comprises logic that generates a network page that comprises the first plurality of search results, wherein the logic that provides the first plurality of search results to the client device sends the generated network page to the client device.

20. The system of claim 16, wherein the application further comprises logic that generates a network page that comprises the second plurality of search results, wherein the logic that provides the second plurality of search results to the client device sends the generated network page to the client device.