tf.contrib.legacy_seq2seq.attention_decoder(
decoder_inputs,
initial_state,
attention_states,
cell,
output_size=None,
num_heads=1,
loop_function=None,
dtype=None,
scope=None,
initial_state_attention=False
)
Defined in tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py
.
RNN decoder with attention for the sequence-to-sequence model.
In this context "attention" means that, during decoding, the RNN can look up information in the additional tensor attention_states, and it does this by focusing on a few entries from the tensor. This model has proven to yield especially good results in a number of sequence-to-sequence tasks. This implementation is based on http://arxiv.org/abs/1412.7449 (see below for details). It is recommended for complex sequence-to-sequence tasks.
Args:
decoder_inputs
: A list of 2D Tensors [batch_size x input_size].initial_state
: 2D Tensor [batch_size x cell.state_size].attention_states
: 3D Tensor [batch_size x attn_length x attn_size].cell
: tf.nn.rnn_cell.RNNCell defining the cell function and size.output_size
: Size of the output vectors; if None, we use cell.output_size.num_heads
: Number of attention heads that read from attention_states.loop_function
: If not None, this function will be applied to i-th output in order to generate i+1-th input, and decoder_inputs will be ignored, except for the first element ("GO" symbol). This can be used for decoding, but also for training to emulate http://arxiv.org/abs/1506.03099. Signature -- loop_function(prev, i) = next- prev is a 2D Tensor of shape [batch_size x output_size],
- i is an integer, the step number (when advanced control is needed),
- next is a 2D Tensor of shape [batch_size x input_size].
dtype
: The dtype to use for the RNN initial state (default: tf.float32).scope
: VariableScope for the created subgraph; default: "attention_decoder".initial_state_attention
: If False (default), initial attentions are zero. If True, initialize the attentions from the initial state and attention states -- useful when we wish to resume decoding from a previously stored decoder state and attention states.
Returns:
A tuple of the form (outputs, state), where:
* outputs
: A list of the same length as decoder_inputs of 2D Tensors of
shape [batch_size x output_size]. These represent the generated outputs.
Output i is computed from input i (which is either the i-th element
of decoder_inputs or loop_function(output {i-1}, i)) as follows.
First, we run the cell on a combination of the input and previous
attention masks:
cell_output, new_state = cell(linear(input, prev_attn), prev_state).
Then, we calculate new attention masks:
new_attn = softmax(V^T * tanh(W * attention_states + U * new_state))
and then we calculate the output:
output = linear(cell_output, new_attn).
* state
: The state of each decoder cell the final time-step.
It is a 2D Tensor of shape [batch_size x cell.state_size].
Raises:
ValueError
: when num_heads is not positive, there are no inputs, shapes of attention_states are not set, or input size cannot be inferred from the input.