CONVOLUTIONAL GATED RECURRENT NEURAL NETWORKS

• US 20170140263A1
• Filed: 11/11/2016
• Published: 05/18/2017
• Est. Priority Date: 11/12/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Sutskever, Ilya, Kaiser, Lukasz Mieczyslaw
• Granted Patent: US 10,977,547 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 17/16 Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a convolutional gated recurrent neural network (CGRN). In one of the systems, the CGRN is configured to maintain a state that is a tensor having dimensions x by y by m, wherein x, y, and m are each greater than one, and for each of a plurality of time steps, update a currently maintained state by processing the currently maintained state through a plurality of convolutional gates.

Claims

1. A system comprising:

a convolutional gated recurrent neural network (CGRN) implemented by one or more computers, wherein the CGRN is configured to;

maintain a state that is a tensor having dimensions x by y by m, wherein x, y, and m are each greater than one,

for each of a plurality of time steps, update a currently maintained state by processing the currently maintained state through a plurality of convolutional gates, and

process the updated state after a last time step in the plurality of time steps through an output layer, wherein the output layer is configured to receive the updated state after the last time step in the plurality of time steps and to modify the updated state to generate a CGRN output.

2. The system of claim 1, wherein the state of the CGRN is initialized to a system input.

3. The system of claim 1, wherein the CGRN comprises a plurality of convolutional gated recurrent units (CGRUs) arranged in a sequence, and wherein for each time step, the lowest CGRU in the sequence is configured to:

receive the currently maintained state and process the currently maintained state to generate a lowest updated state, and wherein each other CGRU in the time step is configured to;

receive the state as updated by a preceding CGRU in the sequence; and

update the received state.

4. The system of claim 3, wherein each CGRU comprises a main convolutional gate, and wherein the main convolutional gate is configured to generate a main convolutional gate output g that satisfies g=σ

(U′

*s+B′

), where s is the state received by the CGRU, U′

*s is a convolution of a kernel bank of the main convolutional gate and the state, and B′

is a bias vector for the main convolutional gate.

5. The system of claim 4, wherein each CGRU further comprises a forget convolutional gate, and wherein the forget convolutional gate is configured to generate a forget convolutional gate output r that satisfies r=σ

(U″

*s+B″

), where s is the state received by the CGRU, U″

*s is a convolution of a kernel bank of the forget convolutional gate and the state, and B″

is a bias vector for the forget convolutional gate.

6. The system of claim 5, wherein the updated internal state as updated by the CGRU CGRU(s) satisfies:

CGRU(s)=u·

s+(1−

u)·

tan h(U*(r·

s)+B)

where U is a kernel bank of the GRU, B is a bias vector, and r·

s signifies pointwise vector multiplication.

7. The system of claim 6, wherein modifying the updated state after the last time step comprises:

applying an output parameter matrix to each entry of a first column of the updated state after the last time step.

8. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:

a convolutional gated recurrent neural network (CGRN) implemented by one or more computers, wherein the CGRN is configured to;

maintain a state that is a tensor having dimensions x by y by m, wherein x, y, and m are each greater than one,

for each of a plurality of time steps, update a currently maintained state by processing the currently maintained state through a plurality of convolutional gates, and

process the updated state after a last time step in the plurality of time steps through an output layer, wherein the output layer is configured to receive the updated state after the last time step in the plurality of time steps and to modify the updated state to generate a CGRN output.

9. The non-transitory computer storage media of claim 8, wherein the state of the CGRN is initialized to a system input.

10. The non-transitory computer storage media of claim 9, wherein the CGRN comprises a plurality of convolutional gated recurrent units (CGRUs) arranged in a sequence, and wherein for each time step, the lowest CGRU in the sequence is configured to:

receive the currently maintained state and process the currently maintained state to generate a lowest updated state, and wherein each other CGRU in the time step is configured to;

receive the state as updated by a preceding CGRU in the sequence; and

update the received state.

11. The non-transitory computer storage media of claim 10, wherein each CGRU comprises a main convolutional gate, and wherein the main convolutional gate is configured to generate a main convolutional gate output g that satisfies g=σ

(U′

*s+B′

), where s is the state received by the CGRU, U′

*s is a convolution of a kernel bank of the main convolutional gate and the state, and B′

is a bias vector for the main convolutional gate.

12. The non-transitory computer storage media of claim 11, wherein each CGRU further comprises a forget convolutional gate, and wherein the forget convolutional gate is configured to generate a forget convolutional gate output r that satisfies r=σ

(U″

*s+B″

), where s is the state received by the CGRU, U″

*s is a convolution of a kernel bank of the forget convolutional gate and the state, and B″

is a bias vector for the forget convolutional gate.

13. The non-transitory computer storage media of claim 12, wherein the updated internal state as updated by the CGRU CGRU(s) satisfies:

CGRU(s)=u·

s+(1−

u)·

tan h(U*(r·

s)+B)

where U is a kernel bank of the GRU, B is a bias vector, and r·

s signifies pointwise vector multiplication.

14. The non-transitory computer storage media of claim 13, wherein modifying the updated state after the last time step comprises:

applying an output parameter matrix to each entry of a first column of the updated state after the last time step.

15. A method comprising:

receiving a system input;

initializing a state using the system input, wherein the state is a tensor having dimensions x by y by m, wherein x, y, and m are each greater than one; and

processing the initialized state using a convolutional gated recurrent neural network (CGRN) implemented by one or more computers, wherein the CGRN is configured to;

for each of a plurality of time steps, update a currently maintained state by processing the currently maintained state through a plurality of convolutional gates, and

process the updated state after a last time step in the plurality of time steps through an output layer, wherein the output layer is configured to receive the updated state after the last time step in the plurality of time steps and to modify the updated state to generate a CGRN output for the system input.

16. The method of claim 15, wherein the CGRN comprises a plurality of convolutional gated recurrent units (CGRUs) arranged in a sequence, and wherein for each time step, the lowest CGRU in the sequence is configured to:

receive the currently maintained state and process the currently maintained state to generate a lowest updated state, and wherein each other CGRU in the time step is configured to;

receive the state as updated by a preceding CGRU in the sequence; and

update the received state.

17. The method of claim 16, wherein each CGRU comprises a main convolutional gate, and wherein the main convolutional gate is configured to generate a main convolutional gate output g that satisfies g=σ

(U′

*s+B′

), where s is the state received by the CGRU, U′

*s is a convolution of a kernel bank of the main convolutional gate and the state, and B′

is a bias vector for the main convolutional gate.

18. The method of claim 17, wherein each CGRU further comprises a forget convolutional gate, and wherein the forget convolutional gate is configured to generate a forget convolutional gate output r that satisfies r=σ

(U″

*s+B″

), where s is the state received by the CGRU, U″

*s is a convolution of a kernel bank of the forget convolutional gate and the state, and B″

is a bias vector for the forget convolutional gate.

19. The method of claim 18, wherein the updated internal state as updated by the CGRU CGRU(s) satisfies:

CGRU(s)=u·

s+(1−

u)·

tan h(U*(r·

s)+B)

where U is a kernel bank of the GRU, B is a bias vector, and r·

s signifies pointwise vector multiplication.

20. The method of claim 19, wherein modifying the updated state after the last time step comprises:

applying an output parameter matrix to each entry of a first column of the updated state after the last time step.