Dynamic tensor attention for information retrieval scoring

• US 10,459,928 B2
• Filed: 12/14/2016
• Issued: 10/29/2019
• Est. Priority Date: 12/14/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Chang, Keng-hao, Zhang, Ruofei, Yin, Zi
• Primary Examiner(s): Beausoliel, Jr., Robert W
• Assistant Examiner(s): Hoang, Hau
• Application Number:
  US15/379,262
• Publication Number:
  US 20180165288A1
• Time in Patent Office: 1,049 Days
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/3325 Reformulation based on results of preceding query
• CPC Class Codes: G06F 16/3346 using probabilistic model
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks

Abstract

A technique of scoring a query against a document using sequence to sequence neural networks. The technique comprises: receiving a query comprising a plurality of words from a user; performing a search for a document comprising words based on the query; feeding the words of the document as the input of an encoder of a multilayer sequence to sequence converter; generating a plurality of vectors at a decoder of the multilayer sequence to sequence converter, each vector comprising a probability associated with a respective word in the query; looking up in the respective vector each word'"'"'s probability of being associated with the document; multiplying every word'"'"'s probability together to determine an overall probability of the query being associated with the document; and returning the document to the user if the overall probability of the query being associated with the document is greater than a threshold value.

Claims

1. A method, comprising:

receiving a query comprising a plurality of words from a user;

performing a search for a document comprising words based on the query;

feeding the words of the document as the input of an encoder of a multilayer sequence to sequence converter;

generating a plurality of vectors at a decoder of the multilayer sequence to sequence converter, each vector of the plurality of vectors being associated with a respective word of the plurality of words in the query;

for each word of the plurality of words in the query, determining from the vector of the plurality of vectors associated with the respective word, a probability of being associated with the document;

multiplying probabilities of being associated with the document for two or more words together to determine an overall probability of the query being associated with the document; and

returning the document to the user if the overall probability of the query being associated with the document is greater than a threshold value.

2. The method of claim 1, further comprising requesting further search information from the user if the overall probability of the query being associated with the document is less than the threshold value.

3. The method of claim 1, wherein the multilayer sequence to sequence converter comprises a three layer sequence to sequence converter.

4. The method of claim 3, further comprising, after feeding the words of the document as the input of an encoder of a multilayer sequence to sequence converter, encoding the words of the document into three hidden vectors h1, h2, and h3.

5. The method of claim 4, wherein generating a plurality of vectors at a decoder further comprises, using the three hidden vectors h1, h2, and h3, along with a previous word in the query to generate the plurality of vectors.

6. The method of claim 5, further comprising using an attention mechanism to generate the probabilities within the plurality of vectors.

7. The method of claim 6, wherein the attention mechanism is a dynamic tensor network.

8. A system comprising:

at least one processor; and

a memory operatively connected with the at least one processor storing computer-executable instructions that, when executed by the at least one processor, causes the at least one processor to execute a method that comprises;

receiving a query comprising a plurality of words from a user;

performing a search for a document comprising words based on the query;

feeding the words of the document as the input of an encoder of a multilayer sequence to sequence converter;

using an attention mechanism to generate a plurality of vectors at a decoder of the multilayer sequence to sequence converter, each vector of the plurality of vectors being associated with a respective word of the plurality of words in the query;

for each word of the plurality of words in the query, determining from the vector of the plurality of vectors associated with the respective words, a probability of being associated with the document, wherein for at least one word in the query, the probability that the at least one word in the query is related to the document is based on a probability that another word in the query is related to the document;

multiplying probabilities of being associated with the document for each word in the query, together to determine an overall probability of the query being associated with the document; and

returning the document to the user if the overall probability is greater than a threshold amount.

9. The system of claim 8, wherein the method further comprises requesting further search information from the user if the overall probability of the query being associated with the document is less than the threshold value.

10. The system of claim 8, wherein the multilayer sequence to sequence converter comprises a three layer sequence to sequence converter.

11. The system of claim 10, wherein the method further comprises, after feeding the words of the document as the input of an encoder of a multilayer sequence to sequence converter, encoding the words of the document into three hidden vectors h1, h2, and h3.

12. The system of claim 11, wherein generating a plurality of vectors at a decoder further comprises, using the three hidden vectors h1, h2, and h3, along with a previous word in the query to generate the plurality of vectors.

13. The system of claim 8, wherein the attention mechanism is a dynamic tensor network.

14. A computer storage media storing computer executable instructions, which, when executed by the computer, cause the computer to perform a method, the method comprising:

receiving a query from a user, the query including a plurality of words;

performing a search on the query;

receiving documents that result from the search;

scoring the documents using a sequence to sequence converter and an attention network to determine a probability that the query is related to the document, wherein the probability that the query is related to the document is based on a probability that a first word of the query is related to the document and a probability that a second word of the query is related to the document, given the first word and the document; and

returning the document if the probability is greater than a threshold value.

15. The computer storage media of claim 14, wherein scoring the documents further comprises inputting the query into an encoder of the sequence to sequence converter.

16. The computer storage media of claim 15, wherein the sequence to sequence converter has a three layer encoder.

17. The computer storage media of claim 16, wherein scoring the documents further comprises generating three hidden vectors, h1, h2, and h3, as outputs of the encoder.

18. The computer storage media of claim 17, wherein a decoder in the sequence to sequence converter receives h1, h2, and h3 as inputs.

19. The computer storage media of claim 14, wherein scoring further comprises determining respective probabilities that each word in the query is associated with the document.

20. The computer storage media of claim 19, wherein the probability is calculated as the product of each of the probability for each word multiplied together.