Neural Networks package in Wolfram Mathematica is not Parallel?

Question

I just created a VERY large neural net, albeit on very powerful hardware, and imagine my shock and disappointment, when I realized that NeuralFit[] from NeuralNetworks` package only seems to use one core, and not even to its fullest capacity. I was heartbroken. Do I really have to write an entire NN implementation from scratch? Or did I miss something simple?

My net took 200 inputs to 2 hidden layers of 300 neurons to produce 100 outputs. I understand we're talking about trillions of calculations, but as long as I know my hardware is the weak point - that can be upgraded. It should handle training of such a net fairly well if left alone for a while (4Ghz 8-thread machine with 24Gb of 2000Mhz CL7 memory running RAID-0 SSD drives on SATA-III - I'm fairly sure).

Ideas? Suggestions? Thanks in advance for your input.

Answer 1

You can contact the author of the package directly, he is a very approachable fellow and might be able to make some suggestions.

Answer 2

I am the author of the Neural Network Package. It is easy to parallelize the evaluation of a neural network given the input. That is, to compute the output of the network given the inputs (and all the weights, the parameters of the network). However, this evaluation is not very time consuming and it is not very interesting to parallellize it for most problems. On the other hand, the training of the network is often time consuming and, unfortunately, not easy to parallelize. The training can be done with a different algorithms and best ones are not easy to parallelize. My contact info can be found at the product's homepage on the Wolfram web. Improvement suggestions are very welcome.

The last version of the package works fine one version 9 and 10 if you switch off the suggestion bar (under preferences). The reason for that is that the package use the old HelpBrowser for the documentation and it crash in combination with the suggestion bar.

yours Jonas

Answer 3

I'm not sure how you wrote the code or how it is written inside the package you are using; try to use vectorization, it really speeds up the linear algebra computations. In the ml-class.org course you can see how it's made.