METHODS AND SYSTEMS FOR SELECTING POTENTIALLY ERRONEOUSLY RANKED DOCUMENTS BY A MACHINE LEARNING ALGORITHM
First Claim
1. A computer-implemented method for selecting a potentially erroneously ranked document in a set of search results, the set of search results having been generated by a search engine server executing a machine learning algorithm (MLA) responsive to a query, the method executable by an electronic device, the electronic device connected to the search engine server, the method comprising:
- receiving, by the electronic device, the set of search results from the search engine server, each document of the set of search results having a relevance score generated by the MLA and a feature vector generated by the MLA, the relevance score having been generated at least in part based on the feature vector;
computing, by the electronic device, for each possible pair of documents of the set of search results, the pair of documents comprising a first document and a second document;
a first parameter obtained by a first binary operation on the relevance scores of the first document and the second document, the first parameter indicative of a level of difference in the relevance scores of the first document and the second document, anda second parameter obtained by a second binary operation on the feature vectors of the first document and the second document, the second parameter indicative of a level of difference in the feature vectors of the first document and the second document;
computing, by the electronic device, a verification score for each possible pair of documents of the set of search results, the verification score being based on first parameter and the second parameter, the verification score indicative of a level of misalignment between the relevance scores of the first document and the second document and the feature vectors of the first document and the second document of the pair of documents;
selecting, by the electronic device, at least one pair of documents associated with an extreme verification score, the extreme verification score indicative of a high level of misalignment between the relevance scores of the first document and the second document and the feature vectors of the first document and the second document of the pair of documents, the high level of misalignment indicative of a possibly erroneously ranked document in the pair of documents; and
marking, by the electronic device, the at least one selected pair of documents associated with the extreme verification score for verification by the search engine server.
4 Assignments
0 Petitions
Accused Products
Abstract
A method and a system for selecting a potentially erroneously ranked document in a set of search results responsive to a query comprising receiving the set of search results from the search engine server, each document of the set of search results having a relevance score and a feature vector generated by an MLA, computing for each possible pair of documents a first parameter indicative of a level of difference in the relevance scores of the documents of the pair of documents and a second parameter indicative of a level of difference in the feature vectors of the documents of the pair of documents, computing a verification score based on first parameter and the second parameter, the verification score indicative of a level of misalignment between the relevance scores and the feature vectors, selecting and marking the pair of documents associated with an extreme verification score for verification.
-
Citations
20 Claims
-
1. A computer-implemented method for selecting a potentially erroneously ranked document in a set of search results, the set of search results having been generated by a search engine server executing a machine learning algorithm (MLA) responsive to a query, the method executable by an electronic device, the electronic device connected to the search engine server, the method comprising:
-
receiving, by the electronic device, the set of search results from the search engine server, each document of the set of search results having a relevance score generated by the MLA and a feature vector generated by the MLA, the relevance score having been generated at least in part based on the feature vector; computing, by the electronic device, for each possible pair of documents of the set of search results, the pair of documents comprising a first document and a second document; a first parameter obtained by a first binary operation on the relevance scores of the first document and the second document, the first parameter indicative of a level of difference in the relevance scores of the first document and the second document, and a second parameter obtained by a second binary operation on the feature vectors of the first document and the second document, the second parameter indicative of a level of difference in the feature vectors of the first document and the second document; computing, by the electronic device, a verification score for each possible pair of documents of the set of search results, the verification score being based on first parameter and the second parameter, the verification score indicative of a level of misalignment between the relevance scores of the first document and the second document and the feature vectors of the first document and the second document of the pair of documents; selecting, by the electronic device, at least one pair of documents associated with an extreme verification score, the extreme verification score indicative of a high level of misalignment between the relevance scores of the first document and the second document and the feature vectors of the first document and the second document of the pair of documents, the high level of misalignment indicative of a possibly erroneously ranked document in the pair of documents; and marking, by the electronic device, the at least one selected pair of documents associated with the extreme verification score for verification by the search engine server. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 19, 20)
-
-
16. A system for selecting a potentially erroneously ranked document in a set of search results, the set of search results having been generated by a search engine server executing a machine learning algorithm (MLA) responsive to a query, the system connected to the search engine server, the system comprising:
-
a processor; a non-transitory computer-readable medium comprising instructions; the processor, upon executing the instructions, being configured to execute; receiving the set of search results from the search engine server, each document of the set of search results having a relevance score generated by the MLA and a feature vector generated by the MLA, the relevance score having been generated at least in part based on the feature vector; computing for each possible pair of documents of the set of search results, the pair of documents comprising a first document and a second document; a first parameter obtained by a first binary operation on the relevance scores of the first document and the second document, the first parameter indicative of a level of difference in the relevance scores of the first document and the second document, and a second parameter obtained by a second binary operation on the feature vectors of the first document and the second document, the second parameter indicative of a level of difference in the feature vectors of the first document and the second document; computing a verification score for each possible pair of documents of the set of search results, the verification score being based on first parameter and the second parameter, the verification score indicative of a level of misalignment between the relevance scores of the first document and the second document and the feature vectors of the first document and the second document of the pair of documents; selecting at least one pair of documents associated with an extreme verification score, the extreme verification score indicative of a high level of misalignment between the relevance scores of the first document and the second document and the feature vectors of the first document and the second document of the pair of documents, the high level of misalignment indicative of a possibly erroneously ranked document in the pair of documents; and marking the at least one selected pair of documents associated with the extreme verification score for verification by the search engine server.
-
Specification