Can I use multiple labels for one feature vector with Lasagne?

Question

I have a semantic segmentation problem where it would be very nice if I could have multiple labels for one feature vector. So I have some parts of my data which belong to class 1, 2 AND 3 (and others which belong to only one class, some to no class at all...).

I think a similar, but much simpler toy problem is building a neural net which gets a number in binary format as input feature and should decide if it is divisible by 2, 3, both, or non of both.

What I've tried

I used nolearn to build a network which has two output neurons (one for "is divisible by 2" and the other for "is divisible by 3".

Please note that I know I could, for this simple example, simply add two classes "is divisible by both" and "is divisible by neither 2 or 3". However, I only created this example for a more complicated problem where I don't have this possibility.

The output layer may not be a softmax layer as I don't want to get 1 in sum of outputs (but 0, 1, or 2). The problem is that I don't know how my label vector should look like. Usually, I only give label_vector = [class_for_first, class_for_second, ...] but this time I need a list of classes. How can I adjust it?

(It does not have to be done with nolearn. A pure Lasagne solution would be fine, too.)

#!/usr/bin/env python

"""Neural Network to decide for numbers in 0..15 if they are divisble by
   2, 3, both or none.
"""

import lasagne
from lasagne import layers
from lasagne.updates import nesterov_momentum
from nolearn.lasagne import NeuralNet

import numpy


def nn_example():
    feature_vectors, labels = [], []
    for i in range(2):
        for j in range(2):
            for k in range(2):
                for l in range(2):
                    feature_vectors.append([i, j, k, l])
                    sum_val = 2**0 * i + 2**1 * j + 2**2 * k + 2**3 * l
                    if sum_val % 2 == 0 and sum_val % 3 != 0:
                        # Only output node for '2' should be one
                        labels.append(0)
                    elif sum_val % 2 != 0 and sum_val % 3 == 0:
                        # Only output node for '3' should be one
                        labels.append(1)
                    elif sum_val % 2 != 0 and sum_val % 3 != 0:
                        # _ALL_ output should be zero
                        labels.append(0)  # dummy value
                    else:
                        # It is divisible by 2 and 3
                        # _BOTH_ output nodes should be 1
                        labels.append(1)  # dummy value
    feature_vectors = numpy.array(feature_vectors, dtype=numpy.float32)
    labels = numpy.array(labels, dtype=numpy.int32)
    net1 = NeuralNet(layers=[('input', layers.InputLayer),
                             ('hidden', layers.DenseLayer),
                             ('hidden2', layers.DenseLayer),
                             ('output', layers.DenseLayer),
                             ],
                     # layer parameters:
                     input_shape=(None, 4),
                     hidden_num_units=3,
                     hidden2_num_units=2,
                     output_nonlinearity=lasagne.nonlinearities.sigmoid,
                     output_num_units=2,

                     # optimization method:
                     update=nesterov_momentum,
                     update_learning_rate=0.01,
                     update_momentum=0.9,

                     max_epochs=1000,
                     verbose=1,
                     )

    # Train the network
    net1.fit(feature_vectors, labels)

    # Try the network
    print("Predicted: %s" % str(net1.predict_proba([[0, 0, 1, 0]])))

if __name__ == '__main__':
    nn_example()

Answer 1

Your labels should be encoded in a matrix (num_samples, num_classes), with all entries either 0 or 1. Take the activation from your sigmoid output layer and compute the cross entropy:

-T.sum(y * T.log(z) + (1 - y) * T.log(1 - z))