EFFICIENT WORD ENCODING FOR RECURRENT NEURAL NETWORK LANGUAGE MODELS

• US 20170091169A1
• Filed: 04/28/2016
• Published: 03/30/2017
• Est. Priority Date: 09/29/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: BELLEGARDA, Jerome R., DOLFING, Jannes G.
• Granted Patent: US 10,366,158 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models

Abstract

Systems and processes for word encoding are provided. In accordance with one example, a method includes, at an electronic device with one or more processors and memory, receiving a user input, determining a first similarity between a representation of the user input and a first acoustic model of a plurality of acoustic models, and determining a second similarity between the representation of the user input and a second acoustic model of the plurality of acoustic models. The method further includes determining whether the first similarity is greater than the second similarity. In accordance with a determination that the first similarity is greater than the second similarity, the first acoustic model may be selected; and in accordance with a determination that the first similarity is not greater than the second similarity, the second acoustic model may be selected.

Claims

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

receive a representation of a current word of a word sequence, wherein the representation of the current word is indicative of a class of a plurality of classes and a word associated with the class;

determine a current word context based on the representation of the current word and a previous word context; and

provide a representation of a next word of the word sequence, wherein the representation of the next word of the word sequence is based on the current word context.

2. The non-transitory computer-readable storage medium of claim 1, wherein the class is a first class and the word is a first word, wherein the representation of the next word of the word sequence is indicative of a second class of the plurality of classes and a second word associated with the second class.

3. The non-transitory computer-readable storage medium of claim 2, wherein the representation of the next word of the word sequence is indicative of the second class of the plurality of classes using a next class value and the representation of the next word of the word sequence is indicative of the second word associated with the second class using a word-within-class value.

4. The non-transitory computer-readable storage medium of claim 1, wherein the representation of the next word is based on a polynomial.

5. The non-transitory computer-readable storage medium of claim 1, wherein the representation of the next word is based on an error correction code.

6. The non-transitory computer-readable storage medium of claim 1, wherein the representation of the current word is encoded using binary code, one or more Gray codes, or a combination thereof.

7. The non-transitory computer-readable storage medium of claim 1, wherein determining a current word context based on the representation of the current word and a previous word context comprises determining the current word context using a hidden layer of a neural network.

8. The non-transitory computer-readable storage medium of claim 1, wherein the representation of the current word input comprises a plurality of words.

9. The non-transitory computer-readable storage medium of claim 1, wherein the representation of the current word and the current word context are associated with a first time step and the previous word context is associated with a second time step.

10. The non-transitory computer-readable storage medium of claim 1, wherein the representation of the current word is a vector representation.

11. The non-transitory computer-readable storage medium of claim 1, wherein the representation of the current word is encoded using 1-of-N encoding.

12. The non-transitory computer-readable storage medium of claim 1,

13. A method comprising:

receiving a representation of a current word of a word sequence, wherein the representation of the current word is indicative of a class of a plurality of classes and a word associated with the class;

determining a current word context based on the representation of the current word and a previous word context; and

providing a representation of a next word of the word sequence, wherein the representation of the next word of the word sequence is based on the current word context.

14. An electronic device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving a representation of a current word of a word sequence, wherein the representation of the current word is indicative of a class of a plurality of classes and a word associated with the class;

determining a current word context based on the representation of the current word and a previous word context; and

providing a representation of a next word of the word sequence, wherein the representation of the next word of the word sequence is based on the current word context.