NEURAL NETWORK DATA ENTRY SYSTEM

• US 20180150744A1
• Filed: 03/30/2017
• Published: 05/31/2018
• Est. Priority Date: 11/29/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Orr, Douglas Alexander Harper, Iso-Sipila, Juha, Fiscato, Marco, Willson, Matthew James, Osborne, Joseph
• Granted Patent: US 10,789,529 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

A data entry system is described which has a user interface which receives a sequence of one or more context text items input by a user. The data entry system has a predictor trained to predict a next item in the sequence. The predictor comprises a plurality of learnt text item embeddings each text item embedding representing a text item in a numerical form, the text item embeddings having a plurality of different lengths. A projection component obtains text item embeddings of the context text items and projects these to be of the same length. The predictor comprises a trained neural network which is fed the projected text item embeddings and which computes a numerical output associated with the predicted next item.

Claims

1. A data entry system comprising:

a user interface which receives a sequence of one or more context text items input by a user;

a predictor trained to predict a next item in the sequence;

the predictor comprising a plurality of learnt text item embeddings each text item embedding representing a text item in a numerical form, the text item embeddings having a plurality of different lengths;

a projection component which obtains text item embeddings of the context text items and projects these to be of the same length;

the predictor comprising a trained neural network which is fed the projected text item embeddings and which computes a numerical output associated with the predicted next item.

2. The data entry system of claim 1 further comprising a scoring component which receives the numerical output of the predictor and computes a plurality of scores of the numerical output with reference to each of a plurality of item embeddings of different lengths, the item embeddings being of text items from a vocabulary.

3. The data entry system of claim 2 wherein the scoring component is configured to compute a dot product of a prefix of the numerical output of the predictor with item embeddings having a length the same as the prefix.

4. The data entry system of claim 2 comprising at least one table of item embeddings, comprising the item embeddings of text items of the vocabulary used by the scoring component, and the text item embeddings of the context items.

5. The data entry system of claim 4 wherein the at least one table of item embeddings is stored in a plurality of separate arrays, one for each item embedding length.

6. The data entry system of claim 4 wherein item embeddings having a same length are stored as adjacent rows of the table of item embeddings.

7. The data entry system of claim 4 wherein the at least one table of item embeddings is stored as a single array in which item embeddings having a same length are stored as adjacent rows of the array and wherein information about which ranges of rows store which lengths of item embedding is also stored.

8. The data entry system of claim 1 wherein the projection component comprises at least two neural networks, one trained to project up item embeddings to a maximum length and one trained to project down item embeddings from the maximum length to a shorter length.

9. The data entry system of claim 8 wherein the projection component comprises two neural networks for each length of item embedding which is shorter than a maximum length of an item embedding.

10. The data entry system of claim 1 wherein the projection component projects the item embeddings of the context items to be the same length by adding zeros to increase the length of some of the item embeddings.

11. The data entry system of claim 1 wherein the projection component projects the item embeddings of the context items by multiplying with a learnt adaptor matrix.

12. The data entry system of claim 11 wherein the projection component comprises a plurality of learnt adaptor matrices, one for each possible item embedding length which is less than a specified maximum.

13. The data entry system of claim 11 wherein the projection component is configured to learn the adaptor matrix as part of training of the neural network.

14. The data entry system of claim 11 wherein the projection component is configured to down project the numerical output of the predictor by reducing its length using the learnt adaptor matrix.

15. The data entry system of claim 1 wherein the projection component is configured to down project the numerical output of the predictor by reducing its length to match that of one or more text item embeddings in a vocabulary used by a scoring component to compute scores of the numerical output.

16. The data entry system of claim 15 wherein the down projection comprises computing a multiplication of the numerical output of the predictor with the adaptor matrix, prior to computing a dot product of the result of the multiplication with an item embedding with a length associated with the adaptor matrix.

17. A computer-implemented method comprising:

receiving a sequence of one or more context text items input by a user;

storing at a memory a plurality of learnt text item embeddings each text item embedding representing a text item in a numerical form, the text item embeddings having a plurality of different lengths;

retrieving text item embeddings of the context text items from the memory and projecting the retrieved text item embeddings to be of the same length; and

inputting the projected text item embeddings to a trained neural network language model and which computes a numerical output associated with a predicted next item of the sequence.

18. The method of claim 17 comprising computing a plurality of scores of the numerical output with reference to each of a plurality of item embeddings of different lengths, the item embeddings being of text items from a vocabulary.

19. The method of claim 18 comprising using a single table at the memory to store both the item embeddings of text items of the vocabulary used by the scoring component, and the text item embeddings of the context items.

20. One or more device-readable media with device-executable instructions that, when executed by a computing system, direct the computing system to perform operations comprising:

receiving a sequence of one or more context text items input by a user;

storing at a memory a plurality of learnt text item embeddings each text item embedding representing a text item in a numerical form, the text item embeddings having a plurality of different lengths;

retrieving text item embeddings of the context text items from the memory and projecting the retrieved text item embeddings to be of the same length; and

inputting the projected text item embeddings to a trained neural network language model and which computes a numerical output associated with a predicted next item of the sequence.