Recurrent layers

The particularity of recurrent is that their output can be fed back as input. This includes common recurrent cells like the Elman RNN or the LSTM.

class dynn.layers.recurrent_layers.ElmanRNN(pc, input_dim, hidden_dim, activation=<function tanh>, dropout=0.0, Whx=None, Whh=None, b=None)

Bases: dynn.layers.base_layers.ParametrizedLayer, dynn.layers.recurrent_layers.RecurrentCell

The standard Elman RNN cell:

\(h_{t}=\sigma(W_{hh}h_{t-1} + W_{hx}x_{t} + b)\)

Parameters:
  • pc (dynet.ParameterCollection) – Parameter collection to hold the parameters
  • input_dim (int) – Input dimension
  • output_dim (int) – Output (hidden) dimension
  • activation (function, optional) – Activation function \(sigma\) (default: dynn.activations.tanh())
  • dropout (float, optional) – Dropout rate (default 0)
__call__(x, h)

Perform the recurrent update.

Parameters:
Returns:

Next recurrent state

\(h_{t}=\sigma(W_{hh}h_{t-1} + W_{hx}x_{t} + b)\)
Return type:

dynet.Expression
__init__(pc, input_dim, hidden_dim, activation=<function tanh>, dropout=0.0, Whx=None, Whh=None, b=None)

Creates a subcollection for this layer with a custom name

get_output(state)

Get the cell’s output from the list of states.

For example this would return h from h,c in the case of the LSTM

init_layer(test=True, update=False)

Initializes only this layer’s parameters (not recursive) This needs to be implemented for each layer

initial_value(batch_size=1)

Return a vector of dimension hidden_dim filled with zeros

Returns:Zero vector
Return type:dynet.Expression
class dynn.layers.recurrent_layers.LSTM(pc, input_dim, hidden_dim, dropout_x=0.0, dropout_h=0.0, Whx=None, Whh=None, b=None)

Bases: dynn.layers.base_layers.ParametrizedLayer, dynn.layers.recurrent_layers.RecurrentCell

Standard LSTM

Parameters:
  • pc (dynet.ParameterCollection) – Parameter collection to hold the parameters
  • input_dim (int) – Input dimension
  • output_dim (int) – Output (hidden) dimension
  • dropout_x (float, optional) – Input dropout rate (default 0)
  • dropout_h (float, optional) – Recurrent dropout rate (default 0)
__call__(x, h, c)

Perform the recurrent update.

Parameters:
Returns:

dynet.Expression for the ext recurrent states

h and c
Return type:

tuple
__init__(pc, input_dim, hidden_dim, dropout_x=0.0, dropout_h=0.0, Whx=None, Whh=None, b=None)

Creates a subcollection for this layer with a custom name

get_output(state)

Get the cell’s output from the list of states.

For example this would return h from h,c in the case of the LSTM

init_layer(test=True, update=False)

Initializes only this layer’s parameters (not recursive) This needs to be implemented for each layer

initial_value(batch_size=1)

Return two vectors of dimension hidden_dim filled with zeros

Returns:two zero vectors for \(h_0\) and \(c_0\)
Return type:tuple
class dynn.layers.recurrent_layers.RecurrentCell(*args, **kwargs)

Bases: object

Base recurrent cell interface

Recurrent cells must provide a default initial value for their recurrent state (eg. all zeros)

__init__(*args, **kwargs)

Initialize self. See help(type(self)) for accurate signature.

__weakref__

list of weak references to the object (if defined)

get_output(state)

Get the cell’s output from the list of states.

For example this would return h from h,c in the case of the LSTM

initial_value(batch_size=1)

Initial value of the recurrent state. Should return a list.

class dynn.layers.recurrent_layers.StackedLSTM(pc, num_layers, input_dim, hidden_dim, dropout_x=0.0, dropout_h=0.0)

Bases: dynn.layers.recurrent_layers.StackedRecurrentCells

Stacked LSTMs

Parameters:
  • pc (dynet.ParameterCollection) – Parameter collection to hold the parameters
  • num_layers (int) – Number of layers
  • input_dim (int) – Input dimension
  • output_dim (int) – Output (hidden) dimension
  • dropout_x (float, optional) – Input dropout rate (default 0)
  • dropout_h (float, optional) – Recurrent dropout rate (default 0)
__init__(pc, num_layers, input_dim, hidden_dim, dropout_x=0.0, dropout_h=0.0)

Initialize self. See help(type(self)) for accurate signature.

class dynn.layers.recurrent_layers.StackedRecurrentCells(*cells)

Bases: dynn.layers.base_layers.BaseLayer, dynn.layers.recurrent_layers.RecurrentCell

This implements a stack of recurrent layers

The recurrent state of the resulting cell is the list of the states of all the sub-cells. For example for a stack of 2 LSTM cells the resulting state will be [h_1, c_1, h_2, c_2]

Example:

# Parameter collection
pc = dy.ParameterCollection()
# Stacked recurrent cell
stacked_cell = StackedRecurrentCells(
    LSTM(pc, 10, 15),
    LSTM(pc, 15, 5),
    ElmanRNN(pc, 5, 20),
)
# Inputs
dy.renew_cg()
x = dy.random_uniform(10, batch_size=5)
# Initialize layer
stacked_cell.init(test=False)
# Initial state: [h_1, c_1, h_2, c_2, h_3] of sizes [15, 15, 5, 5, 20]
init_state = stacked_cell.initial_value()
# Run the cell on the input.
new_state = stacked_cell(x, *init_state)
# Get the final output (h_3 of size 20)
h = stacked_cell.get_output(new_state)
__call__(x, *state)

Compute the cell’s output from the list of states and an input expression

Parameters:x (dynet.Expression) – Input vector
Returns:new recurrent state
Return type:list
__init__(*cells)

Initialize self. See help(type(self)) for accurate signature.

get_output(state)

Get the output of the last cell

initial_value(batch_size=1)

Initial value of the recurrent state.