UNIFIED LANGUAGE MODELING FRAMEWORK FOR WORD PREDICTION, AUTO-COMPLETION AND AUTO-CORRECTION

• US 20170091168A1
• Filed: 04/28/2016
• Published: 03/30/2017
• Est. Priority Date: 09/29/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: BELLEGARDA, Jerome R., DOLFING, Jannes G.
• Granted Patent: US 11,010,550 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

Systems and processes for unified language modeling are provided. In accordance with one example, a method includes, at an electronic device with one or more processors and memory, receiving a character of a sequence of characters and determining a current character context based on the received character of the sequence of characters and a previous character context. The method further includes determining a current word representation based on the current character context and determining a current word context based on the current word representation and a previous word context. The method further includes determining a next word representation based on the current word context and providing the next word representation.

Claims

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

receive a character of a sequence of characters;

determine a current character context based on the received character of the sequence of characters and a previous character context;

determine a current word representation based on the current character context;

determine a current word context based on the current word representation and a previous word context;

determine a next word representation based on the current word context; and

provide the next word representation.

2. The non-transitory computer-readable storage medium of claim 1, wherein the current word representation has a first dimensionality and the next word representation has a second dimensionality greater than the first dimensionality.

3. The non-transitory computer-readable storage medium of claim 1, wherein determining the current word representation comprises:

weighting the current character context using a weight matrix.

4. The non-transitory computer-readable storage medium of claim 1, wherein a dimensionality of the character of the sequence of characters corresponds to a number of characters.

5. The non-transitory computer-readable storage medium of claim 4, wherein a dimensionality of the next word representation corresponds to a number of words.

6. The non-transitory computer-readable storage medium of claim 1, wherein determining the current character context further comprises:

determining the current character context using an activation function, wherein the activation function comprises a sigmoid function, a hyperbolic tangent function, a rectified linear unit function, or a combination thereof.

7. The non-transitory computer-readable storage medium of claim 1, wherein providing the next word representation comprises providing the next word representation using an output layer of a neural network.

8. The non-transitory computer-readable storage medium of claim 7, wherein the neural network is a recurrent neural network.

9. The non-transitory computer-readable storage medium of claim 1, further comprising:

encoding the character of the sequence of characters using 1-of-M encoding binary code.

10. The non-transitory computer-readable storage medium of claim 1, further comprising:

encoding the character of the sequence of characters using one or more Gray codes.

11. The non-transitory computer-readable storage medium of claim 1, further comprising:

encoding the character of the sequence of characters using 1-of-M encoding binary code and one or more Gray codes.

12. The non-transitory computer-readable storage medium of claim 7, wherein at least one of the current word representation and the next word representation are vector representations.

13. A method comprising:

at an electronic device with one or more processors and memory;

receiving a character of a sequence of characters;

determining a current character context based on the received character of the sequence of characters and a previous character context;

determining a current word representation based on the current character context;

determining a current word context based on the current word representation and a previous word context;

determining a next word representation based on the current word context; and

providing the next word representation.

14. An electronic device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving a character of a sequence of characters;

determining a current character context based on the received character of the sequence of characters and a previous character context;

determining a current word representation based on the current character context;

determining a current word context based on the current word representation and a previous word context;

determining a next word representation based on the current word context; and

providing the next word representation.