Neural network data entry system

• US 10,789,529 B2
• Filed: 03/30/2017
• Issued: 09/29/2020
• Est. Priority Date: 11/29/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Orr, Douglas Alexander Harper, Iso-Sipila, Juha, Fiscato, Marco, Wilson, Matthew James, Osborne, Joseph
• Primary Examiner(s): Gonzales, Vincent
• Assistant Examiner(s): Kassim, Imad
• Application Number:
  US15/474,976
• Publication Number:
  US 20180150744A1
• Time in Patent Office: 1,279 Days
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 5/04 Inference or reasoning models

Abstract

A data entry system is described which has a user interface which receives a sequence of one or more context text items input by a user. The data entry system has a predictor trained to predict a next item in the sequence. The predictor comprises a plurality of learnt text item embeddings each text item embedding representing a text item in a numerical form, the text item embeddings having a plurality of different lengths. A projection component obtains text item embeddings of the context text items and projects these to be of the same length. The predictor comprises a trained neural network which is fed the projected text item embeddings and which computes a numerical output associated with the predicted next item.

Claims

1. A data entry system comprising:

a user interface which receives a sequence of one or more context text items input by a user;

processing circuitry configured to execute instructions to implement a plurality of software components, the software components comprising;

a predictor component trained to predict a next item in the sequence;

the predictor component comprising a plurality of text item embeddings stored in a vocabulary, each text item embedding representing a text item in a numerical form, and each text item embedding indicating weights for analysis of a representation of the text item in a trained neural network language model, with the text item embeddings having a plurality of different lengths;

a projection component which obtains text item embeddings for the context text items input by the user, the obtained text item embeddings being of the plurality of different lengths, and projects these to be of the same length;

the predictor component comprising a trained neural network which is fed the projected text item embeddings, having the same length, for use in the trained neural network language model, which computes a numerical output associated with the predicted next item.

2. The data entry system of claim 1 further comprising a scoring component which receives the numerical output of the predictor component and computes a plurality of scores of the numerical output with reference to each of a plurality of item embeddings of different lengths, the item embeddings being of text items from a vocabulary.

3. The data entry system of claim 2 wherein the scoring component is configured to compute a dot product of a prefix of the numerical output of the predictor component with item embeddings having a length the same as the prefix.

4. The data entry system of claim 2 comprising at least one table of item embeddings, comprising the item embeddings of text items of the vocabulary used by the scoring component, and the text item embeddings of the context items.

5. The data entry system of claim 4 wherein the at least one table of item embeddings is stored in a plurality of separate arrays, one for each item embedding length.

6. The data entry system of claim 4 wherein item embeddings having a same length are stored as adjacent rows of the table of item embeddings.

7. The data entry system of claim 4 wherein the at least one table of item embeddings is stored as a single array in which item embeddings having a same length are stored as adjacent rows of the array and wherein information about which ranges of rows store which lengths of item embedding is also stored.

8. The data entry system of claim 1 wherein the projection component comprises at least two neural networks, one trained to project up item embeddings to a predetermined length and one trained to project down item embeddings from the predetermined length to a length shorter than the predetermined length.

9. The data entry system of claim 8 wherein the projection component comprises two neural networks for each length of item embedding which is shorter than the predetermined length of an item embedding.

10. The data entry system of claim 1 wherein the projection component projects the item embeddings of the context items to be the same length by adding zeros to increase the length of some of the item embeddings.

11. The data entry system of claim 1 wherein the projection component projects the item embeddings of the context items by multiplying with an adaptor matrix.

12. The data entry system of claim 11 wherein the projection component comprises a plurality of adaptor matrices, one for each of a plurality of predetermined embedding lengths which are less than a specified maximum.

13. The data entry system of claim 11 wherein the projection component is configured to learn the adaptor matrix as part of training of the neural network.

14. The data entry system of claim 11 wherein the projection component is configured to down project the numerical output of the predictor component by reducing its length using the adaptor matrix.

15. The data entry system of claim 1 wherein the projection component is configured to down project the numerical output of the predictor component by reducing its length to match that of one or more text item embeddings in a vocabulary used by a scoring component to compute scores of the numerical output.

16. The data entry system of claim 15 wherein the down projection comprises computing a multiplication of the numerical output of the predictor component with the adaptor matrix, prior to computing a dot product of the result of the multiplication with an item embedding with a length associated with the adaptor matrix.

17. A computer-implemented method comprising:

receiving a sequence of one or more context text items input by a user;

accessing, from a memory, a plurality of text item embeddings stored in a vocabulary, each text item embedding representing a text item in a numerical form, and each text item embedding indicating weights for analysis of a representation of the text item in a trained neural network language model, with the text item embeddings having a plurality of different lengths;

retrieving, from the memory, text item embeddings for the context text items input by the user, the retrieved text item embeddings being of the plurality of different lengths, and projecting the retrieved text item embeddings to be of the same length; and

inputting the projected text item embeddings, having the same length, for use in the trained neural network language model, which computes a numerical output associated with a predicted next item of the sequence.

18. The method of claim 17 comprising computing a plurality of scores of the numerical output with reference to each of a plurality of item embeddings of different lengths, the item embeddings being of text items from a vocabulary.

19. The method of claim 18 comprising using a single table at the memory to store both the item embeddings of text items of the vocabulary used by the computing of the plurality of scores, and the text item embeddings of the context items.

20. One or more device-readable media with device-executable instructions that, when executed by a computing system, direct the computing system to perform operations comprising:

receiving a sequence of one or more context text items input by a user;

accessing, from a memory a plurality of text item embeddings each text item embedding stored in a vocabulary, representing a text item in a numerical form, and each text item embedding indicating weights for analysis of a representation of the text item in a trained neural network language model, with the text item embeddings having a plurality of different lengths;

retrieving, from the memory, text item embeddings for the context text items input by the user, the retrieved text item embeddings being of the plurality of different lengths, and projecting the retrieved text item embeddings to be of the same length; and

inputting the projected text item embeddings, having the same length, for use in the trained neural network language model, which computes a numerical output associated with a predicted next item of the sequence.