Methods and systems for selecting potentially erroneously ranked documents by a machine learning algorithm

• US 10,642,670 B2
• Filed: 01/04/2018
• Issued: 05/05/2020
• Est. Priority Date: 04/04/2017
• Status: Active Grant

• Current Assignee: Direct Cursus Technology LLC
• Original Assignee: Yandex Europe AG (Yandex N.V.)
• Inventors: Mozhaev, Alexey Vladislavovich
• Primary Examiner(s): Al-Hashemi, Sana A
• Application Number:
  US15/861,993
• Publication Number:
  US 20180285176A1
• Time in Patent Office: 852 Days
• CPC Class Codes: G06F 11/006 Identification G06F11/2289 takes precedence
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

A method and a system for selecting a potentially erroneously ranked document in a set of search results responsive to a query comprising receiving the set of search results from the search engine server, each document of the set of search results having a relevance score and a feature vector generated by an MLA, computing for each possible pair of documents a first parameter indicative of a level of difference in the relevance scores of the documents of the pair of documents and a second parameter indicative of a level of difference in the feature vectors of the documents of the pair of documents, computing a verification score based on first parameter and the second parameter, the verification score indicative of a level of misalignment between the relevance scores and the feature vectors, selecting and marking the pair of documents associated with an extreme verification score for verification.

Claims

1. A computer-implemented method for selecting a potentially erroneously ranked document in a set of search results, the set of search results having been generated by a search engine server executing a machine learning algorithm (MLA) responsive to a query, the method executable by an electronic device, the electronic device connected to the search engine server, the method comprising:

receiving, by the electronic device, the set of search results from the search engine server, each document of the set of search results having a relevance score generated by the MLA and a feature vector generated by the MLA, the relevance score having been generated at least in part based on the feature vector;

computing, by the electronic device, for each possible pair of documents of the set of search results, the pair of documents comprising a first document and a second document;

a first parameter obtained by a first binary operation on the relevance scores of the first document and the second document, the first parameter indicative of a level of difference in the relevance scores of the first document and the second document, and

a second parameter obtained by a second binary operation on the feature vectors of the first document and the second document, the second parameter indicative of a level of difference in the feature vectors of the first document and the second document;

computing, by the electronic device, a verification score for each possible pair of documents of the set of search results, the verification score being based on first parameter and the second parameter, the verification score indicative of a level of misalignment between the relevance scores of the first document and the second document and the feature vectors of the first document and the second document of the pair of documents;

selecting, by the electronic device, at least one pair of documents associated with an extreme verification score, the extreme verification score indicative of a high level of misalignment between the relevance scores of the first document and the second document and the feature vectors of the first document and the second document of the pair of documents, the high level of misalignment indicative of a possibly erroneously ranked document in the pair of documents; and

marking, by the electronic device, the at least one selected pair of documents associated with the extreme verification score for verification by the search engine server.

2. The method of claim 1, wherein the extreme verification score is further indicative of an inability of the MLA of the search engine server to distinguish the first document and the second document of the pair of documents.

3. The method of claim 2, wherein the verification score increases for pairs of document having a high first parameter and a low second parameter, and wherein the verification score decreases for pairs of documents having a low first parameter and a high second parameter.

4. The method of claim 3, wherein the high first parameter is indicative of a high level of difference in the relevance scores of the first document and the second document, and wherein the low first parameter is indicative of a low level of difference in the relevance scores of the first document and the second document.

5. The method of claim 4, wherein the high second parameter is indicative of a high level of difference in the feature vectors of the first document and the second document, and wherein the low second parameter is indicative of a low level of difference in the feature vectors of the first document and the second document.

6. The method of claim 5, wherein each document of the set of search results has a position in a search engine results page (SERP), the position having been determined at least in part based on the relevance score, and wherein the verification score is further based on a third parameter, the third parameter being based on the positions in the SERP of the first document and the second document of the pair of documents.

7. The method of claim 6, wherein the first binary operation is a subtraction and the second binary operation is a subtraction.

8. The method of claim 7, wherein the third parameter is a lowest position between the position of the first document of the pair and the second document of the pair.

9. The method of claim 8, wherein the extreme verification score is a lowest verification score.

10. The method of claim 9, wherein the selecting the at least one pair associated with the lowest verification score comprises selecting a subset of verification scores, each verification score of the subset of verification scores being under a predetermined threshold, the predetermined threshold indicative of an erroneously ranked document.

11. The method of claim 10, wherein the first parameter is weighted by a first coefficient, the second parameter is weighted by a second coefficient and the third parameter is weighted by a third coefficient, the first coefficient, the second coefficient and the third coefficient allowing to reflect a degree of influence of the first parameter, the second parameter and the third parameter in the verification score for selecting the at least one possibly erroneously ranked document.

12. The method of claim 11, wherein the first coefficient, the second coefficient and the third coefficient have been determined heuristically.

13. The method of claim 12, wherein the computing for each pair of adjacent document is performed for a subset of the set of search results.

14. The method of claim 13, further comprising:

transmitting the at least one pair of documents to the search engine server; and

repeating the steps of;

receiving the set of search results, computing the first parameter, the second parameter and the third parameter, computing the verification score, the selecting the at least one pair associated with the lowest verification score and the marking the at least one pair after a predetermined period of time.

15. The method of claim 14, wherein the verification score is computed based on:

16. A system for selecting a potentially erroneously ranked document in a set of search results, the set of search results having been generated by a search engine server executing a machine learning algorithm (MLA) responsive to a query, the system connected to the search engine server, the system comprising:

a processor;

a non-transitory computer-readable medium comprising instructions;

the processor, upon executing the instructions, being configured to execute;

receiving the set of search results from the search engine server, each document of the set of search results having a relevance score generated by the MLA and a feature vector generated by the MLA, the relevance score having been generated at least in part based on the feature vector;

computing for each possible pair of documents of the set of search results, the pair of documents comprising a first document and a second document;

a first parameter obtained by a first binary operation on the relevance scores of the first document and the second document, the first parameter indicative of a level of difference in the relevance scores of the first document and the second document, and

a second parameter obtained by a second binary operation on the feature vectors of the first document and the second document, the second parameter indicative of a level of difference in the feature vectors of the first document and the second document;

computing a verification score for each possible pair of documents of the set of search results, the verification score being based on first parameter and the second parameter, the verification score indicative of a level of misalignment between the relevance scores of the first document and the second document and the feature vectors of the first document and the second document of the pair of documents;

selecting at least one pair of documents associated with an extreme verification score, the extreme verification score indicative of a high level of misalignment between the relevance scores of the first document and the second document and the feature vectors of the first document and the second document of the pair of documents, the high level of misalignment indicative of a possibly erroneously ranked document in the pair of documents; and

marking the at least one selected pair of documents associated with the extreme verification score for verification by the search engine server.

17. The system of claim 16, wherein the extreme verification score is further indicative of an inability of the MLA of the search engine server to distinguish the first document and the second document of the pair of documents.

18. The system of claim 17, wherein the verification score increases for pairs of document having a high first parameter and a low second parameter, and wherein the verification score decreases for pairs of documents having a low first parameter and a high second parameter.

19. The system of claim 18, wherein the high first parameter is indicative of a high level of difference in the relevance scores of the first document and the second document, and wherein the low first parameter is indicative of a low level of difference in the relevance scores of the first document and the second document.

20. The system of claim 19, wherein the high second parameter is indicative of a high level of difference in the feature vectors of the first document and the second document, and wherein the low second parameter is indicative of a low level of difference in the feature vectors of the first document and the second document.