Neural Network Data Sparsity

Question

I am using PyBrain to train a network on music. The input is two notes, and the output is the next two notes. Each note is represented by an integer mapped to a note (E.G C# = 11, F = 7), the octave, and the duration. So I was using a dataset as such:

ds = SupervisedDataSet(6, 6)

Which would look like ([note1, octave1, duration1, note2, octave2, duration2], [note1, octave1, duration1, note2, octave2, duration2])

However, I ran into a problem with chords (I.E more than one note played at once). To solve this, I got rid of the first integer representing a note and replaced it with 22 integers, set to either one or zero, to indicate which notes are being played. I still have this followed by octave and duration. So for example, the following

[0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 4, 0.5]

represents a chord of C#, E and A, with an octave of 4 and duration of 0.5.
PyBrain always gives me an output of all zeros after training and testing. I understand why it's doing this but I don't know how to fix it.
Is there a better way to represent the notes/chords so that PyBrain won't have this problem?

EDIT: I have since converted the bit vector into a decimal number, and while the network isn't just giving zeros anymore it's still pretty clear it's not learning the patterns correctly.

I am using a network like this:

net = buildNetwork(6, 24, 6, bias=True, hiddenclass=LSTMLayer, recurrent=True)

and a trainer like this:

trainer = BackpropTrainer(net, ds, verbose = True)

when I train I am getting a huge error, something like ten or a hundred thousand.

Answer 1

Your problem is not so clear for me, I think it needs more detailed explanation, but depended what I understood I suppose that you don't need reccurence in your network, also try to use another activation function in hidden layer, for example Softmax. I tested it on some data set of samples with 6 nodes input and 6 - output and it is being trained properly, so I there I suggest you my version:

from pybrain.tools.shortcuts import buildNetwork
from pybrain.datasets import SupervisedDataSet
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.structure.modules import SoftmaxLayer

ds = SupervisedDataSet(6, 6)

#
# fill dataset
#
net = buildNetwork(6, 24, 6, bias=True, hiddenclass=SoftmaxLayer)
trainer = BackpropTrainer(net, ds)

train:

error = 10
while error > 0.00001:  #choose error like you want
    error = trainer.train()
    print error #just for logging

#and activate
print net.activate([*,*,*,*,*,*])