Using hidden activations in loss function

Question

I want to create a custom loss function for a double-input double-output model in Keras that:

1. minimizes the reconstruction error of two autoencoders;
2. maximizes the correlation of the bottleneck features of the autoencoders.

For this I need to pass to the loss function:

• both inputs;
• both outputs / reconstructions;
• output of intermediate layers for both (hidden activations).

I know I can pass both inputs and outputs to Model, but am struggling to find a way to pass the hidden activations.

I could create two new Models that have the output of the intermediate layers and pass that to loss, like:

intermediate_layer_model1 = Model(input=input1, output=autoencoder.get_layer('encoded1').output)
intermediate_layer_model2 = Model(input=input2, output=autoencoder.get_layer('encoded2').output)

autoencoder.compile(optimizer='adadelta', loss=loss(intermediate_layer_model1, intermediate_layer_model2))

But still, I would need to find a way to match the y_true in loss to the correct intermediate model.

What is the right way to approach this?

Edit
Here's an approach that I think should work. Simplified:

# autoencoder 1
input1 = Input(shape=(input_dim,))
encoded1 = Dense(encoding_dim, activation='relu', name='encoded1')(input1)
decoded1 = Dense(input_dim, activation='sigmoid', name='decoded1')(encoded1)

# autoencoder 2
input2 = Input(shape=(input_dim,))
encoded2 = Dense(encoding_dim, activation='relu', name='encoded2')(input2)
decoded2 = Dense(input_dim, activation='sigmoid', name='decoded2')(encoded2)

# merge encodings
merge_layer = merge([encoded1, encoded2], mode='concat', name='merge', concat_axis=1)

model = Model(input=[input1, input2], output=[decoded1, decoded2, merge_layer])

model.compile(optimizer='rmsprop', loss={
        'decoded1': 'binary_crossentropy',
        'decoded2': 'binary_crossentropy',
        'merge': correlation,
    })

Then in correlation I can split y_pred and do the calculations.

Answer 1

How about:

1. Defining a single model with a multiple outputs (be sure that you named a coding and reconstruction layer properly):

   duo_model = Model(input=input, output=[coding_layer, reconstruction_layer])
   

2. Compiling your model with two different losses (or even performing a loss reweighting):

   duo_model.compile(optimizer='rmsprop',
                      loss={'coding_layer': correlation_loss, 
                            'reconstruction_layer': 'mse'})
   

3. Taking your final model as a:

   encoder = Model(input=input, output=[coding_layer])
   autoencoder = Model(input=input, output=[reconstruction_layer])
   

After proper compilation this should do the job.

When it comes to defining a proper correlation loss function there are two ways:

1. when coding layer and your output layer have the same dimension - you could easly use predefinied cosine_proximity function from Keras library.
2. when coding layer has different dimensonality - you shoud first find embedding of coding vector and reconstruction vector to the same space and then - compute correlation there. Remember that this embedding should either be a Keras layer / function or Theano / Tensor flow operation (depending on which backend you are using). Of course you can compute both embedding and correlation function as a part of one loss function.