Sentinel gate for modulating auxiliary information in a long short-term memory (LSTM) neural network

• US 10,565,306 B2
• Filed: 11/18/2017
• Issued: 02/18/2020
• Est. Priority Date: 11/18/2016
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Lu, Jiasen, Xiong, Caiming, Socher, Richard
• Primary Examiner(s): Lerner, Martin
• Application Number:
  US15/817,165
• Publication Number:
  US 20180144248A1
• Time in Patent Office: 822 Days
• Field of Search: 704 9, 704232, 706 15, 706 27, 706 30, 706 20
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

The technology disclosed presents a novel spatial attention model that uses current hidden state information of a decoder long short-term memory (LSTM) to guide attention and to extract spatial image features for use in image captioning. The technology disclosed also presents a novel adaptive attention model for image captioning that mixes visual information from a convolutional neural network (CNN) and linguistic information from an LSTM. At each timestep, the adaptive attention model automatically decides how heavily to rely on the image, as opposed to the linguistic model, to emit the next caption word. The technology disclosed further adds a new auxiliary sentinel gate to an LSTM architecture and produces a sentinel LSTM (Sn-LSTM). The sentinel gate produces a visual sentinel at each timestep, which is an additional representation, derived from the LSTM'"'"'s memory, of long and short term visual and linguistic information.

Claims

1. A recurrent neural network system (RNN) running on numerous parallel processors, comprising:

a sentinel long short-term memory network (Sn-LSTM) that;

comprises a memory cell, an input gate, a forget gate, an output gate, and an auxiliary sentinel gate;

receives inputs at each of a plurality of timesteps, the inputs including at least;

an input for a current timestep,

a hidden state from a previous timestep, and

an auxiliary input for the current timestep;

stores in the memory cell auxiliary information accumulated over time from processing of the inputs by the input gate, the forget gate, and the output gate;

updates the memory cell with gate outputs produced by the input gate, the forget gate, and the output gate;

generates, using the output gate, a hidden state as a first output of the Sn-LSTM based on the input, for the current timestep, the hidden state from the previous timestep, and information in the memory cell; and

generates, using the auxiliary sentinel gate, a sentinel state as a second output of the Sn-LSTM different from the first output based on the auxiliary input for the current timestep, the hidden state from the previous timestep, and the information in the memory cell;

wherein the auxiliary sentinel gate modulates the stored auxiliary information from the memory cell for a next prediction, with the modulation conditioned on the input for the current timestep, the hidden state from the previous timestep, and the auxiliary input for the current timestep.

2. The RNN of claim 1, wherein the auxiliary input is visual input comprising image data and the input for the current timestep is a text embedding of a most recently emitted word and/or character.

3. The RNN of claim 1, wherein the auxiliary input for the current timestep is a text encoding from another long short-term memory network (LSTM) of an input document and the input for the current timestep is a text embedding of a most recently emitted word and/or character.

4. The RNN of claim 1, wherein the auxiliary input for the current timestep is a hidden state vector from another LSTM that encodes sequential data and the input for the current timestep is a text embedding of a most recently emitted word and/or character.

5. The RNN of claim 1, wherein the auxiliary input for the current timestep is a prediction derived from a hidden state vector from another LSTM that encodes sequential data and the input for the current timestep is a text embedding of a most recently emitted word and/or character.

6. The RNN of claim 5, wherein the prediction is a classification label embedding.

7. The RNN of claim 1, wherein the auxiliary input for the current timestep is an output of a convolutional neural network (CNN).

8. The RNN of claim 1, wherein the auxiliary input for the current timestep is an output of an attention network.

9. The RNN of claim 1, wherein the Sn-LSTM is further configured to receive multiple auxiliary inputs at a timestep, with at least one auxiliary input comprising concatenated vectors.

10. The RNN of claim 1, wherein the auxiliary input for the current timestep is received only at an initial timestep.

11. The RNN of claim 1, wherein the auxiliary sentinel gate produces a sentinel state at each timestep as an indicator of the modulated auxiliary information.

12. The RNN of claim 11, wherein the sentinel state at the current timestep t is defined as s_t=aux_t ⊙

tan h(m_t), where s_t is the sentinel state, aux_t is the auxiliary sentinel gate applied on the memory cell m_t ⊙

represents element-wise product, and tan h denotes hyperbolic tangent activation.

13. The RNN of claim 1, further configured to use at least the hidden state for the current timestep and the sentinel state for the current timestep for making the next prediction.

14. The RNN of claim 1, wherein the inputs further include a bias input and a previous state of the memory cell.

15. The RNN of claim 1, wherein the Sn-LSTM further includes an input activation function.

16. The RNN of claim 1, wherein the auxiliary sentinel gate generates a pointwise hyperbolic tangent (tan h) of the memory cell.

17. The RNN of claim 1, wherein the auxiliary sentinel gate at the current timestep t is defined as aux_t=σ

(W_x x_t+W_h h_t-1), where W_x and W_h are weight parameters to be learned, x_t is the input for the current timestep, aux_t is the auxiliary sentinel gate applied on the memory cell m_t, and σ

denotes logistic sigmoid activation.

18. A non-transitory, computer-readable medium having computer executable instructions that implement the Sn-LSTM of claim 1.

19. A sentinel long short-term memory network (Sn-LSTM) comprising:

a memory cell for storing a state of the Sn-LSTM;

an output gate that outputs, as a first output of the Sn-LSTM, a current hidden state of the Sn-LSTM based on an input of the Sn-LSTM, a previous hidden state of the Sn-LSTM, and information in the memory cell; and

an auxiliary sentinel gate that;

modulates use of auxiliary information during a next prediction, the auxiliary information being accumulated over time in a memory cell at least from processing of an auxiliary input of the Sn-LSTM combined with the input and the previous hidden state, and

outputs, as a second output of the Sn-LSTM different from the first output, a sentinel state of the Sn-LSTM with the accumulated auxiliary information useful for the next prediction.

20. The Sn-LSTM of claim 19, wherein the auxiliary sentinel gate produces the sentinel state at each timestep as an indicator of the modulated auxiliary information, conditioned on the input for a current timestep, the hidden state from a previous timestep, and the auxiliary input for the current timestep.

21. The Sn-LSTM of claim 19, wherein the auxiliary sentinel gate generates a pointwise hyperbolic tangent (tan h) of the memory cell.

22. A non-transitory, computer-readable medium having computer executable instructions that implement the Sn-LSTM of claim 19.

23. A method comprising:

receiving, at a Sn-LSTM, an input for a current timestep, a hidden state from a previous timestep, and an auxiliary input for the current timestep;

storing, in a memory cell of the Sn-LSTM, auxiliary information accumulated over time from processing of the inputs by an input gate, a forget gate, and an output gate of the Sn-LSTM;

generating, using an output gate of the Sn-LSTM, a hidden state as a first output of the Sn-LSTM based on the input for the current timestep, the hidden state from the previous timestep, and information in the memory cell; and

generating, using an auxiliary sentinel gate of the Sn-LSTM, a sentinel state as a second output of the Sn-LSTM different from the first output based on the auxiliary input for the current timestep, the hidden state from the previous timestep, and the information in the memory cell;

wherein the auxiliary sentinel gate modulates use of auxiliary information during a next prediction, the auxiliary information being accumulated over time in the memory cell at least from the processing of an auxiliary input combined with a current input and a previous hidden state.

24. The method of claim 23, wherein the auxiliary sentinel gate produces the sentinel state at each timestep as an indicator of the modulated auxiliary information, conditioned on the input for a current timestep, the hidden state from the previous timestep, and the auxiliary input for the current timestep.

25. The method of claim 23, wherein the auxiliary sentinel gate generates a pointwise hyperbolic tangent (tan h) of the memory cell.

26. A non-transitory, computer-readable medium having computer executable instructions that carry out the method of claim 23.

27. A computer system running on numerous parallel processors adapted to perform the method of claim 23.

28. A recurrent neural network system (RNN) running on numerous parallel processors for machine generation of a natural language caption for an image, comprising:

an input provider for providing a plurality of inputs to a sentinel long short-term memory network (Sn-LSTM) over successive timesteps, wherein the inputs include at least an input for a current timestep, a hidden state from a previous timestep, and an auxiliary input for the current timestep;

the Sn-LSTM comprising;

at least an input gate, a forget gate, an output gate, and an auxiliary sentinel gate;

a memory cell for storing auxiliary information accumulated over time from processing of the inputs by the input gate and the forget gate;

the output gate for generating, as a first output of the Sn-LSTM, a hidden state of the Sn-LSTM from processing of the input for the current timestep, the hidden state from the previous timestep, and contents of the memory cell;

the auxiliary sentinel gate for modulating the stored auxiliary information from the memory cell to produce, as a second output of the Sn-LSTM, a sentinel state at each timestep, with the modulation conditioned on the input for the current timestep, the hidden state from the previous timestep, and the auxiliary input for the current timestep; and

an emitter for generating the natural language caption for the image based on the sentinel states produced over successive timesteps by the auxiliary sentinel gate.