Why can't I use dropout in the last layer of a lasagne regression model?

Question

A toy regression example. With dropout=0.0 this works fine and the cost decreases. With dropout=0.5 I get the error:

ValueError: Got num_leading_axes=1 for a 1-dimensional input, leaving no trailing axes for the dot product.

Any thoughts?

import theano
import theano.tensor as T
import lasagne
import numpy as np

num_features=10
N=1000

# Set up the network                                                                                                                                                                                                                       
x=T.fmatrix('x')
y=T.fvector('y')

dropout=0.5
network = lasagne.layers.InputLayer(shape=(None, num_features), input_var=x)
if dropout > 0.0:
    network = lasagne.layers.dropout(network, p=dropout),
network = lasagne.layers.DenseLayer( network, num_units=1, nonlinearity=None )

pred = lasagne.layers.get_output(network)

cost = lasagne.objectives.squared_error(pred, y).mean()

# Compile training function                                                                                                                                                                                                                
params = lasagne.layers.get_all_params(network, trainable=True)
train_fn = theano.function([x, y], cost, updates=lasagne.updates.adamax(cost, params) )

# Generate some synthetic data                                                                                                                                                                                                             
X=np.random.randn( N,num_features ).astype( theano.config.floatX )
b=np.random.randn( num_features ).astype( theano.config.floatX )
Y=np.dot( X, b ) + np.random.randn( N ).astype(theano.config.floatX ) * 0.1

# Train for 100 iterations                                                                                                                                                                                                                 
for i in range(100):
    print train_fn(X,Y)

Answer 1

Remove the comma after your dropout layer. The code will work with the dropout immediately after the InputLayer or the DenseLayer. The comma creates a tuple with the network variable (network, ) that causes the error.

network = lasagne.layers.InputLayer(shape=(None, num_features), input_var=x)
if dropout > 0.0:
    network = lasagne.layers.dropout(network, p=dropout),
network = lasagne.layers.DenseLayer( network, num_units=1, nonlinearity=None )