Sequence transduction layers¶

Sequence transduction layers take in a sequence of expression and runs one layer over each input. They can be feed-forward (each input is treated independently, eg. Transduction) or recurrent (the output at one step depends on the output at the previous step, eg. Unidirectional).

class dynn.layers.transduction_layers.Bidirectional(forward_cell, backward_cell, output_only=False)¶

Bases: dynn.layers.base_layers.BaseLayer

Bidirectional transduction layer

This layer runs a recurrent cell on in each direction on a sequence of inputs and produces resulting the sequence of recurrent states.

Example:

# Parameter collection
pc = dy.ParameterCollection()
# LSTM cell
fwd_lstm_cell = dynn.layers.LSTM(pc, 10, 10)
bwd_lstm_cell = dynn.layers.LSTM(pc, 10, 10)
# Transduction layer
bilstm = dynn.layers.Bidirectional(fwd_lstm_cell, bwd_lstm_cell)
# Inputs
dy.renew_cg()
xs = [dy.random_uniform(10, batch_size=5) for _ in range(20)]
# Initialize layer
bilstm.init(test=False)
# Transduce forward
fwd_states, bwd_states = bilstm(xs)
# Retrieve last h
fwd_h_final = fwd_states[-1][0]
# For the backward LSTM the final state is at
# the beginning of the sequence (assuming left padding)
bwd_h_final = fwd_states[0][0]

Parameters:	forward_cell (`recurrent_layers.RecurrentCell`) – The recurrent cell to use for forward transduction backward_cell (`recurrent_layers.RecurrentCell`) – The recurrent cell to use for backward transduction output_only (bool, optional) – Only return the sequence of outputs instead of the sequence of states.

__call__(input_sequence, lengths=None, left_padded=True, output_only=None, fwd_initial_state=None, bwd_initial_state=None)¶

Transduces the sequence in both directions

The output is a tuple forward_states, backward_states where each forward_states is a list of the output states of the forward recurrent cell at each step (and backward_states for the backward cell). For instance in a BiLSTM the output is [(fwd_h1, fwd_c1), ...], [(bwd_h1, bwd_c1), ...]

This assumes that all the input expression have the same batch size. If you batch sentences of the same length together you should pad to the longest sequence.

Parameters:

input_sequence (list) – Input as a list of dynet.Expression objects
lengths (list, optional) – If the expressions in the sequence are batched, but have different lengths, this should contain a list of the sequence lengths (default: None)
left_padded (bool, optional) – If the input sequences have different lengths they must be padded to the length of longest sequence. Use this to specify whether the sequence is left or right padded.
output_only (bool, optional) – Only return the sequence of outputs instead of the sequence of states. Overwrites the value given in the constructor.
fwd_initial_state (dy.Expression, optional) – Overridies the default initial state of the forward recurrent cell.
bwd_initial_state (dy.Expression, optional) – Overridies the default initial state of the backward recurrent cell.

Returns:

List of forward and backward recurrent states: (depends on the recurrent layer)

Return type:

tuple

__init__(forward_cell, backward_cell, output_only=False)¶: Initialize self. See help(type(self)) for accurate signature.

class dynn.layers.transduction_layers.SequenceMaskingLayer(mask_value=0.0, left_padded=True)¶

Bases: dynn.layers.base_layers.BaseLayer

Masks a sequence of batched expressions according to each batch element’s length

This layer applies a mask value to the elements of a sequence of batched expressions which correspond to padding tokens. Typically if you batch a sequence of size 2 and a sequence of size 3 you will pad the first sequence to obtain a list of 3 expresions of batch size 2. This layer will mask the batch element of the last expression corresponding to the padding token in the 1st sequence.

This is useful when doing attention or max-pooling on padded sequences when you want to mask padding tokens with \(-\infty\) to ensure that they are ignored.

Parameters:	mask_value (float, optional) – The value to use for masking left_padded (bool, optional) – If the input sequences have different lengths they must be padded to the length of longest sequence. Use this to specify whether the sequence is left or right padded.

__call__(input_sequence, lengths, left_padded=None)¶

Runs the layer over the input

The output is a list of the output of the layer at each step

Parameters:	input_sequence (list) – Input as a list of `dynet.Expression` objects lengths (list, optional) – If the expressions in the sequence are batched, but have different lengths, this should contain a list of the sequence lengths (default: `None`) left_padded (bool, optional) – If the input sequences have different lengths they must be padded to the length of longest sequence. Use this to specify whether the sequence is left or right padded. Overwrites the value given in the constructor.
Returns:	List of masked expression
Return type:	list

__init__(mask_value=0.0, left_padded=True)¶: Initialize self. See help(type(self)) for accurate signature.

class dynn.layers.transduction_layers.Transduction(layer)¶

Bases: dynn.layers.base_layers.BaseLayer

Feed forward transduction layer

This layer runs one cell on a sequence of inputs and returns the list of outputs. Calling it is equivalent to calling:

[layer(x) for x in input_sequence]

Parameters:	cell (`base_layers.BaseLayer`) – The recurrent cell to use for transduction

__call__(input_sequence)¶

Runs the layer over the input

The output is a list of the output of the layer at each step

Parameters:	input_sequence (list) – Input as a list of `dynet.Expression` objects
Returns:	List of recurrent states (depends on the recurrent layer)
Return type:	list

__init__(layer)¶: Initialize self. See help(type(self)) for accurate signature.

class dynn.layers.transduction_layers.Unidirectional(cell, output_only=False)¶

Bases: dynn.layers.base_layers.BaseLayer

Unidirectional transduction layer

This layer runs a recurrent cell on a sequence of inputs and produces resulting the sequence of recurrent states.

Example:

# LSTM cell
lstm_cell = dynn.layers.LSTM(dy.ParameterCollection(), 10, 10)
# Transduction layer
lstm = dynn.layers.Unidirectional(lstm_cell)
# Inputs
dy.renew_cg()
xs = [dy.random_uniform(10, batch_size=5) for _ in range(20)]
# Initialize layer
lstm.init(test=False)
# Transduce forward
states = lstm(xs)
# Retrieve last h
h_final = states[-1][0]

Parameters:	cell (`recurrent_layers.RecurrentCell`) – The recurrent cell to use for transduction output_only (bool, optional) – Only return the sequence of outputs instead of the sequence of states.

__call__(input_sequence, backward=False, lengths=None, left_padded=True, output_only=None, initial_state=None)¶

Transduces the sequence using the recurrent cell.

The output is a list of the output states at each step. For instance in an LSTM the output is (h1, c1), (h2, c2), ...

This assumes that all the input expression have the same batch size. If you batch sentences of the same length together you should pad to the longest sequence.

Parameters:	input_sequence (list) – Input as a list of `dynet.Expression` objects backward (bool, optional) – If this is `True` the sequence will be processed from left to right. The output sequence will still be in the same order as the input sequence though. lengths (list, optional) – If the expressions in the sequence are batched, but have different lengths, this should contain a list of the sequence lengths (default: `None`) left_padded (bool, optional) – If the input sequences have different lengths they must be padded to the length of longest sequence. Use this to specify whether the sequence is left or right padded. output_only (bool, optional) – Only return the sequence of outputs instead of the sequence of states. Overwrites the value given in the constructor. initial_state (`dy.Expression`, optional) – Overridies the default initial state of the recurrent cell
Returns:	List of recurrent states (depends on the recurrent layer)
Return type:	list

__init__(cell, output_only=False)¶: Initialize self. See help(type(self)) for accurate signature.