SENTINEL LONG SHORT-TERM MEMORY (Sn-LSTM)

• US 20180144248A1
• Filed: 11/18/2017
• Published: 05/24/2018
• Est. Priority Date: 11/18/2016
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: LU, Jiasen, XIONG, Caiming, SOCHER, Richard
• Granted Patent: US 10,565,306 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

The technology disclosed presents a novel spatial attention model that uses current hidden state information of a decoder long short-term memory (LSTM) to guide attention and to extract spatial image features for use in image captioning. The technology disclosed also presents a novel adaptive attention model for image captioning that mixes visual information from a convolutional neural network (CNN) and linguistic information from an LSTM. At each timestep, the adaptive attention model automatically decides how heavily to rely on the image, as opposed to the linguistic model, to emit the next caption word. The technology disclosed further adds a new auxiliary sentinel gate to an LSTM architecture and produces a sentinel LSTM (Sn-LSTM). The sentinel gate produces a visual sentinel at each timestep, which is an additional representation, derived from the LSTM'"'"'s memory, of long and short term visual and linguistic information.

Claims

1. A recurrent neural network system (abbreviated RNN) running on numerous parallel processors, comprising:

a sentinel long short-term memory network (abbreviated Sn-LSTM) that

receives inputs at each of a plurality of timesteps, the inputs including at least

an input for a current timestep,

a hidden state from a previous timestep, and

an auxiliary input for the current timestep;

generates outputs at each of the plurality of timesteps by processing the inputs through gates of the Sn-LSTM, the gates including at least

an input gate,

a forget gate,

an output gate, and

an auxiliary sentinel gate;

stores in a memory cell of the Sn-LSTM auxiliary information accumulated over time from

processing of the inputs by the input gate, the forget gate, and the output gate, and

updating of the memory cell with gate outputs produced by the input gate, the forget gate, and the output gate; and

the auxiliary sentinel gate modulates the stored auxiliary information from the memory cell for next prediction, with the modulation conditioned on the input for the current timestep, the hidden state from the previous timestep, and the auxiliary input for the current timestep.

2. The RNN of claim 1, wherein the auxiliary input is visual input comprising image data and the input is a text embedding of a most recently emitted word and/or character.

3. The RNN of claim 1, wherein the auxiliary input is a text encoding from another long short-term memory network (abbreviated LSTM) of an input document and the input is a text embedding of a most recently emitted word and/or character.

4. The RNN of claim 1, wherein the auxiliary input is a hidden state vector from another LSTM that encodes sequential data and the input is a text embedding of a most recently emitted word and/or character.

5. The RNN of claim 1, wherein the auxiliary input is a prediction derived from a hidden state vector from another LSTM that encodes sequential data and the input is a text embedding of a most recently emitted word and/or character.

6. The RNN of claim 5, wherein the prediction is a classification label embedding.

7. The RNN of claim 1, wherein the auxiliary input is an output of a convolutional neural network (abbreviated CNN).

8. The RNN of claim 1, wherein the auxiliary input is an output of an attention network.

9. The RNN of claim 1, wherein the Sn-LSTM is further configured to receive multiple auxiliary inputs at a timestep, with at least one auxiliary input comprising concatenated vectors.

10. The RNN of claim 1, wherein the auxiliary input is received only at an initial timestep.

11. The RNN of claim 1, wherein the auxiliary sentinel gate produces a sentinel state at each timestep as an indicator of the modulated auxiliary information.

12. The RNN of claim 11, wherein the outputs comprise at least a hidden state for the current timestep and a sentinel state for the current timestep.

13. The RNN of claim 12, further configured to use at least the hidden state for the current timestep and the sentinel state for the current timestep for making the next prediction.

14. The RNN of claim 1, wherein the inputs further include a bias input and a previous state of the memory cell.

15. The RNN of claim 1, wherein the Sn-LSTM further includes an input activation function.

16. The RNN of claim 1, wherein the auxiliary sentinel gate gates a pointwise hyperbolic tangent (abbreviated tan h) of the memory cell.

17. The RNN of claim 1, wherein the auxiliary sentinel gate at the current timestep t is defined as aux_t=σ

(W_xx_t+W_hh_t-1), where W_x and W_h are weight parameters to be learned, x_t is the input for the current timestep, aux_t is the auxiliary sentinel gate applied on the memory cell m_t, ⊙

represents element-wise product, and σ

denotes logistic sigmoid activation.

18. The RNN of claim 11, wherein the sentinel state at the current timestep t is defined as s_t=aux_t⊙

tan h (m_t), where st is the sentinel state, aux_t is the auxiliary sentinel gate applied on the memory cell m_t, ⊙

represents element-wise product, and tan h denotes hyperbolic tangent activation.

25. A non-transitory, computer-readable medium having computer executable instructions that implement the Sn-LSTM and the auxiliary sentinel gate of claim 1.

19. A sentinel long short-term memory network (abbreviated Sn-LSTM), running on numerous parallel processors, that processes auxiliary input combined with input and previous hidden state, comprising:

an auxiliary sentinel gate that

applies on a memory cell of the Sn-LSTM, and

modulates use of auxiliary information during next prediction, the auxiliary information accumulated over time in the memory cell at least from the processing of the auxiliary input combined with the input and the previous hidden state.

20. The Sn-LSTM of claim 19, wherein the auxiliary sentinel gate produces a sentinel state at each timestep as an indicator of the modulated auxiliary information, conditioned on an input for a current timestep, a hidden state from a previous timestep, and an auxiliary input for the current timestep.

21. The Sn-LSTM of claim 19, wherein the auxiliary sentinel gate gates a pointwise hyperbolic tangent (abbreviated tan h) of the memory cell.

26. A non-transitory, computer-readable medium having computer executable instructions that implement the Sn-LSTM and the auxiliary sentinel gate of claim 19.

22. A method, including:

extending a long short-term memory network (abbreviated LSTM) to include an auxiliary sentinel gate that

applies on a memory cell of the LSTM, and

modulates use of auxiliary information during next prediction, the auxiliary information accumulated over time in the memory cell at least from the processing of auxiliary input combined with current input and previous hidden state.

23. The method of claim 22, wherein the auxiliary sentinel gate produces a sentinel state at each timestep as an indicator of the modulated auxiliary information, conditioned on an input for a current timestep, a hidden state from a previous timestep, and an auxiliary input for the current timestep.

24. The method of claim 22, wherein the auxiliary sentinel gate gates a pointwise hyperbolic tangent (abbreviated tan h) of the memory cell.

27. A non-transitory, computer-readable medium having computer executable instructions that carry out the method of claim 22.

28. A computer system running on numerous parallel processors adapted to perform the method of claim 22.

29. A recurrent neural network system (abbreviated RNN) running on numerous parallel processors for machine generation of a natural language caption for an image, comprising:

an input provider for providing a plurality of inputs to a sentinel long short-term memory network (abbreviated Sn-LSTM) over successive timesteps, wherein the inputs include at least an input for a current timestep, a hidden state from a previous timestep, and an auxiliary input for the current timestep;

a gate processor for processing the inputs through each gate in a plurality of gates of the Sn-LSTM, wherein the gates include at least an input gate, a forget gate, an output gate, and an auxiliary sentinel gate;

a memory cell of the Sn-LSTM for storing auxiliary information accumulated over time from processing of the inputs by the gate processor;

a memory cell updater for updating the memory cell with gate outputs produced by the input gate, the forget gate, and the output gate;

the auxiliary sentinel gate for modulating the stored auxiliary information from the memory cell to produce a sentinel state at each timestep, with the modulation conditioned on the input for the current timestep, the hidden state from the previous timestep, and the auxiliary input for the current timestep; and

an emitter for generating the natural language caption for the image based on the sentinel states produced over successive timesteps by the auxiliary sentinel gate.