Transposed convolution on feature maps using Theano

Question

I asked similar question on CrossValidation for the image interpretation. I'm moving my detailed question here to include some code details.

The results I'm having are not fully desirable So maybe you have faced this issue before and you can help me find it out.

It is fully convolution neural network "no fully connected part".

Training part

first the images are transposed to match the convolution function. (batch_no,img_channels,width,height)

input.transpose(0, 3, 1, 2)

Learning optimized using learning rate:3e-6, Hu_uniform initialization and nestrove for 500 epochs until this convergence.

• Training cost: 1.602449
• Training loss: 4.610442
• validation error: 5.126761
• Test loss: 5.885714

Backward part

Loading Image

jpgfile = np.array(Image.open(join(testing_folder,img_name)))

Reshape to one batch

batch = jpgfile.reshape(1, jpgfile.shape[0], jpgfile.shape[1], 3)

Run the model to extract first feature map after activation using Relu

output = classifier.layer0.output
    Test_model = theano.function(
        inputs=[x],
        outputs=output,
    )

    layer_Fmaps = Test_model(test_set_x)

Apply backwork model to reconstruct the image using the only activated neurons

bch, ch, row, col = layer_Fmaps.shape
output_grad_reshaped = layer_Fmaps.reshape((-1, 1, row, col))
output_grad_reshaped = output_grad_reshaped[0].reshape(1,1,row,col)
input_shape = (1, 3, 226, 226)
W = classifier.layer0.W.get_value()[0].reshape(1,3,7,7)
kernel = theano.shared(W)
inp = T.tensor4('inp')
deconv_out = T.nnet.abstract_conv.conv2d_grad_wrt_inputs(
            output_grad = inp,
            filters=kernel,
            input_shape= input_shape,
            filter_shape=(1,3,7,7),
            border_mode=(0,0), 
            subsample=(1,1)
        )
f = theano.function(
            inputs = [inp],
            outputs= deconv_out)

f_out = f(output_grad_reshaped)
deconved_relu =  T.nnet.relu(f_out)[0].transpose(1,2,0)
deconved =  f_out[0].transpose(1,2,0)

Here we have two images results, the first is the transposed image without activation and the second with relu since kernels might have some negative weights.

It is clear from the transposed convolution image that this kernel is learn to detect some useful feature related to this image. But the reconstructing part is breaking the image color scheme during the transpose convolution. It might be because the pixels values are small float numbers. Do you see where is the problem here ?